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ABSTRACT 


Technical personnel are increasingly being required to 
perform vital functions as proposal evaluators in the source 
selection process for which they have not properly been trained. 
This resource effort provides a comprehensive system for source 
selection using price and other factors in a form aimed at the 
technical professionals that support field acquisition activi- 
ties. All examples selected are from the general acquisition 
area of service contracting. The system consists of the basic 
considerations necessary for preparation of a procurement 
request, the basic elements of a source selection plan, selec- 
tion of a technical evaluation panel, selection of evaluation 
factors for service contractors, preparation of negotiation 
objectives, the actual conduct of evaluations and negotiations, 


and debriefing of unsuccessful offerors. 
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LT INTRODUCTION 


A. GENERAL 

In the past ten years Navy technical personnal have been 
increasingly required to be contract administrators and 
proposal evaluators. This fact has been clearly recognized 
at major technical activities such as the Naval Weapons Center, 
China Lake. Senior contract specialist positions have been 
established in several of the major technical departments to 
provide liaison with the acquisition personnel and to provide 
guidance in contract administration and proposal evaluation 
J1].. However, most activities supported by the Navy Field 
Acquisition System (NFAS) cannot afford the luxury of providing 
technical personnel with contracting advisors. Nevertheless, 
the technical personnel are expected to perform as vital mem- 
hers of the contracting team. 

The Department of Defense does provide a one week course 
on "Defense Contracts Management for Technical Personnel" that 
covers the complete acquisition life cycle [2:l]>- Since a 
Significant number of technicians and engineers are unable to 
attend this course, the Navy goes one step further and provides 
a one week course for "Contracting Officer's Technical Repre- 
sentative (COTR)." The COTR course was specifically developed 
to train technical personnel in contract administration and is 


often given on site. However, it lightly touches the area of 
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proposal evaluation [3:II-5]. Interviews with NFAS personnel 
confirm that a void exists in the training of technical per- 


sonnel in proposal evaluation [4]. 


B. STATEMENT OF THE PROBLEM 
Poorly planned, defined, conducted, and documented pro- 
posal evaluations have contributed to the following: 


a. contractor dissatisfaction resulting in disputes, 
protests and claims [5:3-49]. 


b. qualified contractors deciding not to submit pro- 
posals or submitting inadequate proposals [5:3-49]. 


c. long procurement administrative leadtimes [6:I-l]. 
d. frustration of technical personnel [6:1]. 
e. selection of unsatisfactory contractors (5:3-49]. 


f. difficulty in defending selection of contractors to 
the General Accounting Office (GAO) [5:3-50]. 


fe RESEARCH OBJECTIVE 

The objective of this research effort is to develop a 
comprehensive system for the source selection of service 
contractors using price and other factors. This source 
@emection system is to be developed with the technical members 
of the evaluation panel in mind. 

Service contracts were selected because individuals inter- 
Viewed reported service contracts as the most common area to 
use evaluation factors other than price. It is important to 
note that this research effort does not address modes of evalu- 
ation. It will address the planning, defining, conducting, 
aggregating, documenting, and use of evaluation efforts in the 


source selection process. 
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D. RESEARCH QUESTION 

What constitutes a thorough and impartial source selection 
system using price and other factors that is suitable for use 
as a standard in service contracting within the Navy Field 


Acquisition System? 


BE. SCOPE, LIMITATIONS, AND ASSUMPTIONS 
i «oCope 

A broad definition of service contracts is used to 
include any requirement for a contractor's time and effort 
rather than for a concrete end item. Examples of contracting 
actions that typically would require proposal evaluation 
using price and other factors are: 

a. expert and consultant services. 


b. services for the operation of Government owned 
facilities and systems. 


Cc. engineering and technical services. 
d. research and development (R&D) services. 


e. automatic data processing equipment (ADPE) 
services. 


2. Limitations 
Contracts for services that are acquired using small 
purchase procedures are specifically excluded. Major weapons 
systems related service acquisitions are excluded because the 
proposal evaluations are often supported by complex, computer 


models. 
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3. Assumptions 
It is assumed that the reader has a technical back-~ 
ground and only a minimal understanding of Government 


Semceracting. 


F. METHODOLOGY 
1. Literature Search 

A comprehensive search was conducted of the literature 
base for applicable studies and articles in the areas of 
source selection and decision and policy analysis. Informa- 
tion was obtained primarily from the Naval Postgraduate School 
library and the Defense Logistics Information Center, Fort 
Lee, VA. 

2. Legal Search 

The primary legal search method was a detailed review 
of the Quarterly Digests of Unpublished Decisions of the 
Comptroller General of the United States and the published 
Decisions of the Comptroller of the Unites States. A second- 
ary method was a computer search using key words of all GAO 
decisions. This service was provided by the Federal Legal 
Information Through Electronics, Denver, CO. The legal 
search concentrated on GAO decisions due to the pre-award 
nature of proposal evaluations. GAO decisions are issued in 
response to protests submitted by contractors to appeal the 
actions of a contracting officer. Very seldom do contractors 
appeal to the courts concerning proposal evaluations. A com- 


puter search of federal court decisions was very unproductive. 
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Major Activity Source Selection Guides 


An effort was made to collect as many major activity 


source selection guides, handbooks or instructions as possible. 


Examples of documents collected are: 


Oakland 


Beach. 


ae 


De 


U. S. Department of Energy, "Procurement 
Regulation Handbook," 30 June 1979. 


National Aeronautics and Space Administration, 
“Source Evaluation Board Manual," change 2, 
25 January 1980. 


U. S. Army Material Development and Readiness 
Command, “Contract Management Guidance for 
Technical Personnel," advanced copy. 


“Source Selection Process Handbook for the 
Air Force Space Division," May 1980. 


Naval Air Development Center, Warminster, 
"The Preparation and Processing of Purchase 
Requests,” undated. 


Personal Inverviews 


Finally, visits were made to the Naval Supply Center, 


and to the Naval Regional Contracting Office, Long 


Personal interviews were conducted of contract 


Specialists, management personnel and legal counsel. The 


purpose of the interviews was to identify and evaluate pro- 


posal evaluation techniques and procedures currently being 


used. 
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II. SOURCE SELECTION METHODS 


Nothing in contract management is more important than 
an informed start. Created by early communications between 
informed personnel, it greatly improves the probability of 
an uneventful procurement. The usual result is a shortened 
cycle for the award of a contract and a greater potential 
for satisfactory performance by a thoroughly evaluated 


Semntractor. [6:1] 


A. BASIC CONSIDERATIONS 

An informed start begins with the consideration of the 
full spectrum of source selection methods on each procure- 
ment. The most basic consideration is whether the procure- 
ment is sole source or competitive. This is a formal 
determination that must be approved by the contracting 
officer. The complexity of the service being acquired often 
dictates a need for more complex source selection procedures. 
The time available for specification development and procure- 
ment action may limit the options available. The availability 
of personnel that might be part of the Government evaluation 
team may be a constraint. A very real limitation may be the 
availability of sources in private industry that are capable 
Of providing the required services. Of course, a vital con- 
Sideration is the estimated dollar value of the procurement 


which may control the cost-benefit of investing expensive 
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administrative effort into source selection. This list of 
basic considerations is not comprehensive and is provided only 


as a- start. 


B. SOURCE SELECTION METHODS 
ime Lype.or ConrEract 

In acquiring services where technical or engineering 
competency is not the controlling factor, source selection 
in terms of price alone may be sufficient. Award may be made 
to the lowest responsive and responsible offeror and a pre- 
award survey may be relied upon to assist in these determina- 
tions [2:18]. A small business may appeal a negative 
responsibility determination to the Small Business Administra- 
tion (SBA) which may issue a Certificate of Competency. This 
document for all intent and purposes is a conclusive determina- 
tion of positive responsibility. 

When to use evaluation factors in service contracting 
is a subject of much heated debate and contradictory opinions. 
One extreme view is that evaluation factors are appropriate 
for all service contracts because a special skill can always 
be identified. Examples given were contracts for moving and 
storage [7] and for mess attendant contracts [8]. A much 
more conservative view was taken by legal counselt interviewed 
who expressed a concern for the use of evaluation factors on 
any fixed price contracts [9]. GAO has provided some guidance 
On these questions and has consistently held that evaluation 


factors may be used in source selection on fixed price contracts 
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and especially on cost type contracts. Particularly, when 
ne Government "is seeking creativity and innovation rather 
than manpower thus making factors other than price paramount" 
mo}. 

The fact is that a service requirement may lend itself 
to the use of a fixed price contract. If a suitable contrac- 
tor can be obtained by considering pricing alone, then consi- 
derable saving can be obtained in contractor proposal costs, 
Government administrative cost and time. For that reason 
the Naval Regional Contracting Office, Long Beach, has been 
instrumental in developing a new contract type currently being 
used in mess attendant contracts. The Fixed Price, Indefinite 
Quantity with Award Amount (FP(IQ)-W/AA) type contract was 
developed to counter the negative effects of heavy competition 
without the use of evaluation factors. It recognizes the fact 
that the bulk of the contract is for manpower that has very 
Beetle variation from contractor to contractor. The contractor 
1s required to pay employees at least the prevailing wage rate 
as set by the U. S. Department of Labor (DOL) and seldom does 
a contractor pay more. Therefore, the FP(IQ)-W/AA type con- 
tract establishes a fixed service rate based on the DOL wage 
determination and the contractor invoices for each service 
unit delivered. For mess attendant contracts, the service 
unit is manhours of food service attendant services delivered. 

All management functions are grouped under a separate 


item entitled "Management and Support Price." Award is made 
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to the responsive and responsible bidder that submits the 
lowest bid for Management and Support Price. A separate 
award mechanism is used to motivate the contractor to provide 
quality service. At no time are evaluation factors used in 
the source selection process. 

However, desire to use a fixed price contract does 
not mean that evaluation factors are not appropriate. Educa-~ 
tional services are a common example. Required expertise is 
clearly identifiable and finite end products can be estab- 
lished on a per course or per student basis. [7] 

Zee QUualificatign Factors 

Another method of source selection is the use of 
special standards of responsibility or qualification factors. 
These are go-no go factors such as the geographical location 
of facilities, degree of licensing requirements of key per- 
sonnel or specific experience in a specialized field of oper- 
ations [2:17]. Qualification factors should be objectively 
verifiable and judgment should not be a significant 
requirement [ll]. 

Qualification factors may be used alone or in conjunc- 
tion with evaluation factors. When used with evaluation fac- 
tors, they should be set out separately and clearly identified. 
Mixing qualification factors with evaluation factors may delay 
the start of procurement action [6:I-l]. The use of qualifi- 
cation factors in this manner serves to notify potential con- 


tractors of the minimum qualifications and resources necessary 
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to perform the proposed work of a ivemeprocusement:. OLferors 
who do not possess these requirements should not be encouraged 
to incur proposal and other expenses involved in a competitive 
submission. [12:4-2] 
3. Evaluation Factors 

In the evaluation-factor method, the Government activi- 
ties have broad latitude in determining the evaluation factors 
and the proposal evaluation method to be utilized. The only 
requirements are that the evaluation method provide a rational 
basis for source selection and that the evaluation itself be 
conducted in good faith and in accordance with the announced 
evaluation factors in the RFP. Most NFAS activities utilize 
numerical point ratings in an attempt to quantify what is 
essentially a subjective judgment. However, neither GAO nor 
procurement regulations require that proposals be evaluated 
on the basis of numerical scores. [13] 

There are other source selection methods such as two 
step formal advertising which are rarely used in service 


Semtracting [ll]. 


C. THE SOURCE SELECTION PLAN (SSP) 

A formal document that is seldom used by NFAS activities, 
at least by this title, but which is commonly used in major 
weapons systems acquisition, is the Source Selection Plan 
(SSP) [14:7]. In short, a SSP is the Government's statement 


to itself as to how it intends to purchase what it wants. 
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Most NFAS activities do not require such a document because 
most procurement actions do not have a complex source selec- 
tion procedure and the benefits do not warrant the added 
administrative burden [li]. However, several individuals 
interviewed agreed that the use of a SSP would be warranted 
for source selection on high dollar value service contracts. 
The extent and detail of the SSP would increase with the 
complexity and potential for protest [15]. 

The SSP is a major component of the total acquisition 
plan and therefore must be taken into consideration during 
the earliest steps of the acquisition process. In service 
contracting, the other major component is the statement of 
work (SOW). Ideally, the SOW and SSP should be developed 
concurrently. This is the first example of the need for early 
information flow and teamwork between the technical and con- 
tracting personnel. The need for teamwork certainly does not 
end here, but must extend through the entire source selection 
process. Teamwork is the most vital prerequisite to the 
success of any acquisition. [2:45] 

1. Basic Elements 
The basic elements of the SSP are: [16:3-3] 
= a brief description of the acquisition. 


- identification of the innovation or expertise 
required. 


—- proposed composition of the technical evaluation 
panel (TEP). 


- proposed evaluation factors and weights. 
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- rating or scoring technique. 

- intent to use preproposal conference. 

- summary of independent cost estimate (ICE). 

—- proposed milestones. 

2. Preproposal Conference 

Preproposal conferences are a valuable tool and they 
are often used by NFAS activities [ll]. They serve to provide 
valuable background material and to clarify complex SOW and 
RFP provisions. SOW and RFP inconsistencies may be surfaced 
and corrected prior to proposal submission. The result can be 
a Significant savings in time and administrative effort as 
compared to individual sequential visits. It is important 
that all Government personnel be completely briefed prior to 
the conference and an accurate record be made of the conference 
proceedings. [16:3-19] 

3. Independent Cost Estimate (ICE) 

Careful consideration should be made towards the devel- 
opment of an ICE. There is a definite relationship between 
the Government's ability to develop an accurate ICE and the 
problems which will be encountered later in the source selec- 
tion process. An ICE is important because it will enhance the 
identification of potential funding problems as well as "buy- 
ins." A "buy-in" is where the offeror purposely underestimates 
on a cost type contract and plans on subsequent cost overruns 
should he be awarded the contract [16:3-8]. GAO has made 


attempts to require that an ICE be developed for every 
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acquisition that may require a detailed cost/price analysis 
[17]. However, an ICE is expensive and time consuming to 
prepare and NFAS activities do not prepare an ICE for every 
acquisition [7]. 

4, Source Selection Plan Milestones 

Reasonable milestones should be established for the 
pealowing critical events: 

- issuance of the RFP. 

- receipt of proposals. 

- competitive range determination. 

— receipt of Best and Final (B&F). 

- contract award. 

It is important that sufficient time be given to both 
Government personnel and contractor personnel for each step 
in the contracting process. Undue compression of the con- 
tracting process can discourage competition, restrict innova- 
tion, and increase errors in proposals and evaluations [8]. 
Naval Regional Contracting Office, Long Beach, studies have 
shown that the most time consuming effort is the evaluation 
by the requiring activity. Also, there have been extensive 
delays in issuing the RFP and in the receipt of proposals due 


to RFP amendments [ll]. Again, teamwork is the key to meeting 


the milestone dates. 
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D. STATEMENT OF WORK 

A statement of work (SOW) is a document that describes 
accurately the nonspecification requirements of a job or 
work effort including the standards used to determine whether 
the requirements have been met [18:4]. The function of a 
SOW is to articulate, as straightforwardly and unambiguously 
as possible, the requirements that will lead to the perform- 
ance of work that achieves job or work effort objectives 
[6:II-l]. This is a tough and slippery area and unless one 
understands the nature of services contracting it can be 
troublesome [6:III-1l1]. 

This research effort does not include the development 
of a SOW. Detailed guidance may be obtained in the Department 
of Defense military handbook MIL-HDBK=-245A(NAVY) dated 
1 August 1978 entitled "Preparation of Statement of Work 
(SOW) ." Also, Office of Federal Procurement Policy Pamphlet 
No. 4 entitled "A Guide to Writing and Administering Perform- 
ance Statements of Work for Service Contracts" provides 
guidance. 

f. Lmpact 

The SOW plays a critical role throughout the entire 
source selection process. It impacts the type of contract 
selected, the decision of offerors to propose, the credibility 
of proposed cost or price, the usefulness of the SSP, and the 
level of resentment of unsuccessful offerors [6:II-2]. A 


Clearly written SOW will allow offerors to understand the 
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Government requirement and respond with detailed plans and 
approaches for accomplishment of the tasks specified. It 
will allow offerors to make reasonable estimates of the 
resources which will be required to satisfy the Government's 
requirements. A poorly written SOW is misleading and presents 
a poor image to private industry [19:28]. The RFP may be 
delayed as contracting personnel and technical personnel 
attempt to upgrade the SOW. The receipt of proposals may be 
delayed due to extensive offeror questions and resulting 
amendments to the RFP. Finally, the evaluation and award may 
be delayed due to wide variations in proposals and lengthy 
Megetrations to clarify the requirement. Often, a contractor 
may build in costly contingency allowances if a clear and 
definite SOW is not provided [5:3-15]. A well defined service 
Can be acquired using a fixed price or incentive type contract, 
while the inadequately defined service often uses time and 
material or cost type contracts. 
2. Matching SOW and SSP 

It is extremely important that the SOW and SSP be 
GCaretully matched to each other. Normally, the first two 
steps taken by a prospective offeror in examining an RFP is 
to read the SOW and proposed evaluation factors [6:IV-2]. 
The most common mismatch is when the SOW clearly describes 
a normal work effort that can be performed by personnel trained 
to one level of expertise, but the evaluation factors are 


established for a worst case situation and indicate a much 
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higher level of expertise. This has become a very common 
area for protest by unsuccessful offerors. Although this 
type of protest is very difficult for an offeror to win, it 
does cause delays and considerable Government administrative 
Beeoert and frustration [15]. 

If there is a mismatch between the SOW and the evalu- 
ation factors, an experienced contractor will direct his 
proposal towards the evaluation factors [ll]. If the SOW is 
inherently spongy, the contractor may direct his proposal 
specifically at the evaluation factors, win the award and then 
take off in pursuit of expanding the scope of work during 
performance or cutting back on the work effort [6:II-3]. 

This can lead to considerable follow on contract administration 
problems. : 
Pee ating tekesSOW to a Stardard 

A standard is defined as an acknowledged measure of 
comparison. A standard in order to be useful must be measur- 
able. A SOW must not only describe a particular work effort, 
but must aescribe that work in terms of standards that con- 
Sistently establish a particular quality level. The evalua- 
tion factors are also selected, defined, and weighted to 
Match the same quality level. Each proposal is evaluated 
against the standards of the SOW and not against each other. 
This provides consistency, objectivity and comparability 


Dpetween proposals. [18:4] 
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E. TECHNICAL EVALUATION PANEL (TEP) 
1. Establishing a TEP 

Normally, in an acquisition where source selection is 
based on an evaluation with primary emphasis on factors 
other than price, a technical evaluation panel (TEP) is estab- 
lished to perform the actual evaluation and to provide a 
recommendation to the contracting officer. The panel gener- 
ally consists of at least three members and seldom more than 
five. Some major activities have local restrictions against 
the use of only one member on the TEP although not specifi- 
cally forbidden by procurement requlations. A TEP chief is 
selected to chair the meetings and to be a central contact 
point. The vote of the TEP chief should be equal to the other 
panel members [11]. Ideally, the TEP should be identified at 
least by organization involved and position, if not by name, 
prior to submitting the procurement request (PR) to the con- 
Beaeting activity [2:7]. 

In selecting TEP members consideration should be given 
to their particular areas of expertise and to their avail- 
ability to perform the evaluation. It is possible that some 
TEP members may evaluate proposals in all areas and others may 
evaluate only a single area related to their own expertise. 
TEP members are usually from the requiring activity but evalu- 
ators may be used from any Government activity. Nongovernment 
personnel should not be used as TEP voting members but may be 
used on an advisory/consultant basis if they are acting in an 


feeereciadl Capacity or under contract for that purpose. If 
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consultants are to be used, they can be listed in the RFP to 
surface any conflicts of interests prior to initial receipt 
of proposals [15]. Consultants should be allowed access only 
to those portions of the proposal that are necessary to enable 
them to give specific technical advise [16:3-5]. 
Zee so ctandards of Conduct 

TEP members for major weapons systems acquisition are 
required to sign a statement that they wiil make no unauthor- 
ized disclosure or release of any source selection informa- 
tion. A second signature is required on a statement that they 
have no personal conflicts of interest with any potential 
offeror. Sample statements are provided as Figures 1 and 2 
mee331). 

The general rules of conduct are: [14:31] 


- direct all inquiries pertaining to the source 
selection to the contracting officer. 


= do not permit office personnel to divulge TEP 
member activity to casual callers. 


- do not assume that a non-participating contractor 
can be told anything pertaining to the source 
selection activity. 


- do not discuss proposals among TEP members at 
social events. 


- CONG EmacCCept any TmVvitatLon Erom contractor 
personnel to participate in any social affair 
regardless of how remote it may be to the TEP 
ae Clee. 


- do not discuss TEP sensitive data even after the 
announcement of a winning contractor. 
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Disclosure of Source Selection Information Statement 


I certify that I will make no unauthorized disclosure 
or release of any source selection information. This 
includes the contents of all proposals submitted in 
response to (contract number), and any evaluation 
thereof. I further certify that I will not discuss 
with any unauthorized person the panel's finding or 
decisions, the contractor's approaches, or any data 
generated during the selection process. I understand 
that unauthorized disclosure or release of any source 
selection information may subject me to disciplinary 
or adverse administrative action. 


Figure l 


CGimimMcte OL Laeerest Statement 


I certify that neither I nor my immediate family, to 
the best of my knowledge, possesses any financial 
interest whatsoever in any company, parent or subsi- 
diary, which is proposing or quoting on or is any way 
inyolved in the acquisition now being considered by 
the source selection board of which I am a member. 
Should any company in which I or my immediate family 
have a financial interest submit quotes or proposals 
to my board, I will reveal immediately such interest 
to the chairperson of the source selection board. 


Figure 2 
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The degree of technical sophistication as well as the 
dollar value and sensitivity of the acquisition are among 
those considerations that dictate the magnitude of the TEP 
briefing effort. NFAS activities have on occasion required 
TEP members to sign disclosure and conflict of interest state- 
ments but more often they send a cover letter addressing the 
subject along with the initial proposals [ll]. Whatever 
the manner, information security can be improved by an early 
briefing of the TEP. Another important factor is holding to 
a firm schedule. As time drags on there is a greater chance 
of a leak [7]. 

3. Basic Duties 

The basic duties of the TEP are: [5:2-8] 

= review and finalize the SOW. 

- recommend evaluation criteria. 

= recommend sources for solicitation. 

- evaluate technical proposals. 

- participate in preproposal conferences. 


- prepare technical documentation in support 
Or CONntracting Officer. 


- participate in contract negotiations. 

- participate in debriefing unsuccessful offerors. 

When proposals are submitted to the TEP for evalua- 
tion, the members should be advised that the evaluation must 
be conducted in strict accordance with the evaluation factors 


set forth in the RFP. They should also be advised that the 
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evaluation documentation must contain a reasonably complete 
discussion of each proposal under each factor [20:13-1]. 

The members should be made aware of the Freedom of Information 
Act (FOIA) and that the evaluation documentation may eventu- 
ally be made available to the offerors. In many ways the 

FOIA has been a benefit in obtaining better evaluation 

reports [ll]. This subject is addressed in greater detail in 
Gmapter VI. 

The TEP evaluations are to be based on all available 
information including proposals, discussions, reference and 
other appropriate checks, and the personal knowledge of the 
individual members in the areas of their expertise [12 73=3) . 
The TEP should establish an internal procedure for resolving 
individual differences of Opinion and negotiating a group 
result. Differences may also develop between the TEP and 
the contracting officer. The TEP should be aware that the 
decisions of the contracting officer are final and the TEP 
Must act in accordance with the decision [5:2-10]. analy , 
the TEP should keep in mind that the Government may be force 
to bear the burden of a defective evaluation. This may mean 
anything from an expression of displeasure from GAO to a 
directive that the current contract be cancelled and resoli- 
Cited [9]. There have been some cases where the TEP evalu- 
ations failed to give a fair and honest consideration to 
the offeror's proposal and the offeror was entitled to recover 


proposal preparation costs from the TEP activity [21]. 
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III. MULTI-ATTRIBUTE UTILITY MEASUREMENT (MAUM) 


Procuring activities may exercise considerable discretion 
in determining the particular method of proposal evaluation 
to be utilized. The only requirements are that the method 
provide a rational basis for source selection and that the 
evaluation itself be conducted in good faith [13]. However, 
there are entire fields of study directed at identifying and 
improving evaluation techniques and procedures. Decision 
analysis is a discipline for systematic evaluation of alter- 
native actions in order to make a choice among them [22:viil]. 
Some other related fields are systems analysis and policy 
analysis [23:4]. Extensive research and field testing has 
been conducted to identify those evaluation techniques and 
procedures that are most effective. 

A sub-division of decision analysis that has been gaining 
popularity in the last five years is Multi-Attribute Utility 
Measurement (MAUM). MAUM is an explicit technique that is a 
decision making tool. The MAUM procedure for evaluating 
Multiple attributes of various alternatives is essentially 
the same process as NFAS activities currently use in source 
selection. Multiple evaluation factors are established and 
weighted and each proposal is scored using these factors by 
a technical panel of experts. There are few guidelines on 
the proper use of evaluation factors being used by NFAS 
activities. The explicit nature of MAUM would bring a sense 


of direction and exactness to the source selection process 


p24). 
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In a very pure definition, MAUM is not a method of evalu- 
ation, but rather a widely applicable method for organizing 
and presenting evaluation information [25:1]. After the 
evaluation factors are clearly identified, ranked and 
weighted, the individual evaluator is still faced with the 
problem of how to evaluate the proposal for that particular 


mactor. 


i. PURPOSE 

The purpose of this chapter is to describe a version of 
MAUM that is simple and straightforward so that any TEP that 
wishes to conduct a MAUM evaluation may do so without refer- 
ence to any other source. There are some rather sophisticated 
versions of MAUM currently being used by major weapons systems 
activities that require trained analyst and computer support. 
It is assumed that the TEP members at most NFAS activities 
are not trained analysts even though they are recognized 
S=pertsS in a particular field. Therefore, it is further 
assumed that the TEP would reject any overly complex analyti- 


cal procedure. 


B. SIMPLE MULTI-ATTRIBUTE RATING TECHNIQUES (SMART) 

The Simple Multi-Attribute Rating Technique (SMART) is a 
version of MAUM specifically developed for the non-professional 
analyst. The technique was developed primarily by Dr. Ward 
Edwards, currently the Director, Social Science Research 


institute, University of Southern California. It has been used 
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primarily in the evaluation of social programs such as 
criminal justice programs and school busing plans. Because 
of the nature of social programs, the primary input to their 
evaluation is the judgment of experts in the field. In 
contrast, the physical characteristics of the hardware is a 
major portion of a weapons system evaluation. Superior 
properties can be determined by measurement and testing. 
Because of the nature of service contracting efforts, the 
primary input to their evaluation is also the judgment of 
experts. For this reason, this researcher believes that 
all the field testing of SMART on evaluation of social pro- 
grams establishes its validity for use in the evaluation of 
service contractors. 

The following description of SMART is a paraphrase of 
written material provided by Dr. Edwards as supplemented by 
a personal interview. The primary difference is that the 
nomenclature has been converted from analytical language to 
procurement language. 

The basic assumptions of SMART are [25:1-3] 


- requests for services normally have multiple 
goals which are not equally important. 


- judgments are inevitably a part of any evaluation. 


- judgments of magnitude are best when expressed 
numerically. 


- evaluations are directly relevant to the source 
selection decision. 
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Because a source selection must be defensible, evaluators 
may be hesitant to use judgment to arrive at a recommendation. 
However, unless the judgment of the TEP is clearly without a 
reasonable basis, or there is an abuse of discretion, or a 
violation of procurement statutes, the judgment of the TEP 
will not be disturbed [26]. Nevertheless, if the TEP uses an 
intuitive method of scoring an evaluation factor when indus- 
try standards are available, they could expect strong criti- 
cism because it is expected they will use hard facts and 
data when available [27:11]. 

Sometimes expert evaluators are reluctant to express 
their judgments in numerical form. However, Dr. Edwards 
States, "...the evidence abundantly indicates that numerically 
expressed judgments are far more accurate and far more useful 
than human judgment expressed in the more familiar quantita- 
tive verbal forms." [28:181] A sound narrative is still 
required, but the numbers lend themselves to convenient manip- 
ulation, aggregation, and trade off evaluation. 

Dr. Edwards suggests three steps to be used to increase 
the likelihood that the evaluation will be used in the 
decision [25:1-10]. 


_ involve the contracting officer or source 
selection official in the evaluation process. 


—- make the evaluation directly relevant to the 
source selection decision. 


- make the product of the evaluation readable and 
easy to use. 
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C. ELEMENTS OF SMART SYSTEM 

The SMART system is characterized by a ten step procedure. 
The basic technique does not change, only the identity and 
expertise of the typical evaluators. | 

1. Step One: Identify the person, persons, organization, 
or organizations whose values are to be maximized. 

Both persons and organizations will usually be rele- 
vant generic "stakeholders." [28:189] A typical series of 
questions to ask are: 

- Who are the end users of the service? 

- Who are the potential contractors? 

- Who is the contracting officer? 


- What is the potential for SBA, Congressional 
or other high level involvement? 


2. Step Two: What is the purpose of the evaluation? 

This is more than simply source selection. The ulti- 
mate purpose is to provide a service in order to fill an 
identified requirement. 

3. Step Three: Identify the entities to be evaluated. 

A feature of the evaluation process for source selec- 
tion is that the evaluation factors must be set forth in the 
RFP and, therefore, before receipt of proposals. Although 
not normally required by the SMART technigue, it does improve 
the objectivity of the evaluation [24]. However, it does 
require substantial prior knowledge of potential contractors 


and their essential characteristics. 





It is important to decide whether the evaluation will 

be based solely on the technical content of the proposals or 
‘be expanded to include the contractors themselves. The 
evaluation of a contractor requires the use of responsibility 
type factors such as “experience” and "facilities." It is 
improper to use only responsibility type factors. However, 
it is proper to use responsibility type factors in conjunction 
with evaluation factors bearing on technical approach when 
used to make relative assessments of the merits of individual 
proposals. A separate responsibility finding is made on the 
contractor in line for award after the proposal evaluation 
has been completed [10]. 
4. Step Four: Identify the relevant evaluation factors. 

The motto of decision analysis is "dividé and conquer." 
25-31) This means that the evaluation problem should be 
divided into separate elements and then recombined by means 
Of an appropriate formal aggregation rule. In addition, the 
judgmental tasks should be partitioned to fit the expertise of 
the evaluators. 

This evaluation factor structuring task is probably 
the most important, difficult and creative aspect of SMART. 
The procedure recommended by Dr. Edwards is to develop an 
evaluation factor tree. A hierarchical structure of evaluation 
factors with more abstract and unmeasurable factors at the top 
and better defined and more measurable sub-factors at the 
Bottom [28:191]. (See figure 3.) The advantages of the tree 


structure are: [25:2-8] 
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Step Four 


Identify the relevant evaluation factors 


Factor A 
Factor Bl 
PaSeee Factor B2 
Factor B3 
\ Raceormec lL 
Factor C2 
Faccome C Factor C3 
Factor C4 
Hactom C5 
BactoL Di 


Factor D Factor D2 


Factonm D3 


Pacwine 3 
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- PePOTesencsmrdctorc in a more Orderly structure, 
thereby helping though about the problem. 


= it can help reduce the number of judgments 
required. 


= it permits subaggregation. 

Experience shows that there is a tendency for individ- 
uals to add factors while seldom will anyone eliminate a 
factor. The result often is too many evaluation factors. 
Ideally, only four or five factors are enough. However, since 
many more factors than that must usually be considered, 
Dr. Edwards recommends that eight are enough and fifteen are 
definitely too many. The reasons why the number of evaluation 
factors should be kept small are: [28:19] 

_ a large number of factors dilutes the 

importance of an important factor relative 


to another. 


~ more work is required to gather necessary 
INnponmtaci On . 


_ more difficult to maintain independence 
between factors. 


Dr. Edwards recommends several methods for reducing 
the number of factors: ie 92 | 


- assemble the TEP face-to-face to improve 
communications. 


_ look for factors that are simply the relabeling 
of other factors. 


- look for factors that have a high environmental 
correlation 


- eliminate the factors that are not important 
enough to influence the decision. 
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An example of environmental correlation would be in 
the evaluation of senior academic personnel to provide 
training services. There would be a high correlation between 
the degree status and the number of published writings. One 
should either combine the factors and evaluate together or 
select one factor to use as a proxy for the other [24]. 

5. Step Five: Rank the factors in relative order of 
importance. 

For evaluation factors arranged in a tree, factors 
beneath each node in the tree are ranked separately. This 
means the major factors would be ranked with each other and 
the sub-factors would be ranked separately under each major 
factor. (See figure 4.) 

6. Step Six; Assign weights toeach evaluation factor. 

Weights are useful because all the factors in the 
evaluation tree are not of equal importance. Weights serve 
as devices that represent the relative importance of each 
factor to all others. They establish an exchange rate among 
evaluation factors. The SMART technique allows the use of 
Several weighting procedures, but Dr. Edwards recommends the 
“ratio method." [24] 

TO assign weights using the ratio method, start with 
the least important major factor and assign a weight of 10. 
Then compare this factor with the next most important factor 
and aSSign a weight that depicts how much more important that 


Factor is relative to the least important factor. For example, 
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Step Five 


Rank the factors in relative order of importance 


Factor C4 
Factor C3 
Factor e Paceor C5 
Raceor. CL 


Factor C2 


Factor Dl 
Factor D Factor D3 
Baceop — DZ 
PaccOnss5 
Factor B Factor B2 
Factor Bl 


Factor A 


Figure 4 
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in figure 4, factor B is twice as important as factor A and 
therefore receives a weight of 20. In working up the list 
of major factors, the TEP must consider not only the least 
important factor, but all other factors that have already 
been assigned weights. Constant adjustments of weights are 
to be expected. After the major factors are completed, 
each individual group of sub-factors is assigned weights 
using exactly the same ratio method starting with a weight 
of 10. Ties are permitted but not encouraged because they 
complicate full disclosure in the RFP [25:4-6]. 

7. Step Seven: Normalize the weights and calculate the 
final weight assignments. 

Normalization is a simple mathematical technique used 
to make all the factor weights add up to 100 and for each 
group of sub-factors to add up to the weight assigned the 
Major factor. The weights assigned the major factors are 
Summed. The weight assigned each major factor is divided by 
the sum of the major factors and multiplied by 100 to give the 
Final weight. For example, in figure 5, the final weight for 
Factor C is calculated dividing the original weight of factor 
C (50). by the sum of all original weights (115) and multiplying 
by 100 to get 43.478. Obviously, any mathematical technique 
that starts with judgmental inputs is not accurate to three 
decimal places. Therefore, each TEP should discuss this issue 
and come to a consensus in order to maintain comparability 


between the evaluations of each panel member. For this example, 
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Step Six 


Assign weights to each evaluation factor 


c4 60 
es 45 
C 30 cs 40 
er 20 
C2 10 
Dl 30 
ee D3 20 
D2 10 
B3 5 
B 20 B2 es 
Bl 10 
A 10 
Figure 5 
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the final weight of factor C is rounded to 43.5. Each group 
of sub-factors is normalized in exactly the same manner, 
except that instead of multiplying by 100, muitiply by the 
Major factor weight. (See figure 6.) 

A side benefit of this step is that the folly of 
including too many evaluation factors becomes apparent. If 
a factor has a final weight less than or approaching 1.0, it 
may be trivial. One should consider combining this factor 
with another factor or deleting the factor altogether [29:11]. 

8. Step Eight: Scoring and rating the proposals. 

A score or a rate is an assessment of how desirable 
a particular proposal is with respect to a particular factor. 
Prior to the receipt of proposals, a scoring or rating 
method should be established for all judgmental factors. 
This will aid each TEP member in scoring and increase the 
comparability between their scores. A common technique is to 
establish a scale that relates adjective statements with 
numerical values. This method is typically used in personnel 
evaluations. [30:12] 

In developing a scoring or rating scale, several 
points should be considered: [25:5-5] 


= evaluators are reluctant to approach the 
extremes, especially the lower extremes. 


- the scale should serve to differentiate one 
proposal from another and therefore should 
be spread out. 


= the scale should allow for the evaluation 
of proposals outside the expected range. 
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Step Seven 


Normalize the weights and calculate the final assignment 


7 or xx100=43.5 


35 = 
D Tpsx100=30.4 


20 2 
B Trpexlo0=17.4 


10 7 
Trex100=8.7 


Factor sum = 


100 


C4 


cl 


DL 


DS 


D2 


B3 


Be 


Bl 


60 _ 
P75x43.5=14.9 


45 : 
p7eX43.5q1l.2 


raex43.5= 9.9 


oa5x43 .5= 5.0 


10 7 
T7g%43..5= eS 


Sum oe 


30 _ 
=px30.4=15.2 


20 _ 
$px30-4=10.2 


10 i: 
Bo%x30.4= 550 


52x17.4= 8.7 


25 
soxl7.4= Ge2 


50.4 


27.45 


Figure 6 
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There is a close relationship between the scoring or 
rating scale and the weights assigned in Step Six. Separate 
scales can be developed for each evaluation factor based on 
the expected range of proposals. Within the expected range, 
an acceptable zone is identified bounded by the minimum accep- 
table and maximum acceptable. For ease of computations and 
with little loss of accuracy the scale within the acceptable 
zone is normally assumed to be linear. The weights assigned 
are good only for proposals in this acceptable zone [25:5-12]. 
In the evaluation of social programs the weights may be 
adjusted if the alternatives are different than original 
expectations. However, because DAR does not allow major 
changes in weights after the receipt of proposals, mechanisms 
should be designed into the scoring or rating scale 
in order to consider proposals outside the original acceptable 
zone. 

From a practical standpoint, 4a universal scoring or 
rating system can be used instead of developing a scale for 
each factor. Alterations are then limited to asjusting the 
weights. An example of a universal scoring or rating scale 
is provided below. Notice that the evaluator does not have to 
know the weight assigned a particular factor in order to score 
feae factor. 

1.0 Exceeds specified performance or capability and 


excess is useful (not gold plating). 


47 





PmeOutctanding — 9a dertinite Strength that meets all 
the requirements of the RFP with a high probability of success. 

.8 Above average - a minor asset but with room for 
improvement. 

.7 Average - neither a strength or weakness with 
Siomaricant room for improvement. 

.6 Below average - a minor deficiency but acceptable 
without correction. 

Q.0 Unsatisfactory - fails to meet the standards of 
the RFP and is not correctable without a major rewrite of the 
proposal. 

This universal scoring or rating scale has a mechanism 
for evaluating excess above the acceptable range and a mechan- 
ism for making unacceptable proposals stand out numerically. 
It allows excess to be considered which encourages innovation 
and alternative proposals. Excess is set apart and allowed to 
enter into the computations only if the excess is usefui. 

The wide spread between Weak and Meee seactory is important 
because it forces the evaluators to address the critical 
question whether the deficiency is capable of being corrected. 
If a major factor is unacceptable, the entire proposal is 
Unacceptable. Numerically, a zero score on a major factor 
would clearly set a proposal apart from other proposals that 
had scored at least fifty percent in all major factors. 

The mathematics of using this scale are illustrated 


by figure 7. 
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Step Eight 


Scoring and rating the proposals 


GA 14.9x.9=13.4 
@3 li.2x.6=6.8 
Cye43.5 Gon 9eoeeus= 7.9 
ei 35.0x.75=3.8 


G2 PA AS > epg — ua are 


Delo. e seo) 221 


Proposal X/D 30.4 > EO 2 2 9 5=9'27 


Dew oO x O= 5.0 


B3 Be vxeo> 7.0 
B 17.4 Boe Os esac- 5.0 
Bien. ke i= 18 


Peo nl. B= 71 0 
Max possible = 100 


Figure 7 
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9. Step Nine: Aggregate. 

The aggregation rule is very simple and is illustrated 
by figure 8. The larger the final numerical score, the better 
the proposal. Normally, the proposal with the largest numeri- 
cal value would be considered the best proposal [25:6-2]. 

10. Step Ten: Decide. 

It is important to realize that SMART is only a 
decision making tool and the results are not the decision [33]. 
The results must be presented to the contracting officer or 
the source selection official in a manner that objectively 
presents and supports the recommendation of the TEP. Important 
trade-offs and options should be identified. In complex, high 
dollar value procurements, a sensitivity analysis may require 
computational support. Dr. Edwards has developed programs for 
the TI-59 and HP=-41C hand held programmable calculators which 


are available upon request. 
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BZ. 1 


S557 


Zion. 6 


14.6 


Step Nine 


ngeee Gace 
C4 13.4 
G3 650 
E> ae 
cl Bio 
CZ ero 
D1 Zed 
D3 eee. 
D2 re 
B3 aS 
B2 iO 
Bak lees: 

Figure 8 
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IV. GOVERNMENT CONSTRAINTS 


Although procuring activities have broad latitude in the 
source selection process, there are a considerable number of 
constraints under which they must operate. Surprisingly, the 
major body of these constraints does not come from the Defense 
Acquisition Regulations, but from the decisions of the Comp- 
troller General of the United States. These decisions are a 
result of protests filed by contractors or the result of formal 
requests by administrative agencies for a GAO decision. Once 
the decision is made and issued, it is binding on the procur- 
ing activity. Decisions of general interest are published 


monthly in pamphlets and annually in volumes. [31:13] 


A. SUMMARY OF THE RULES 

The Ee wiing is a summary of the rules as extracted from 
the GAO decisions and other literature by the researcher con- 
cerning the source selection process. It is by no means 
complete and any specific question should be referred to legal 
counsel. 

1. Procuring activities have broad latitude in determining 
the particular method of proposal evaluation [13]. 


- The method must provide a rational basis for 
source selection. 


= The evaluation must be conducted in good faith. 


- Numerical scoring is not required. 


De 





wer 6A 1 
informed of 


importance. 


3. GAO 


GAO will not disturb unless shown to be arbitrary 
or contrary to statutes or regulations. 


offerors in a negotiated procurement must be 


the major evaluation factors and their relative 


The weights of the evaluation factors shall not 
be disclosed [32]. 


If one factor is assigned a predominant value, 
its greater relative importance shall be indicated 
by appropriate language [33]. 


If no indication of the relative importance of 
factors or sub-factors is given, offerors may 
assume equal importance [34]. 


Any minimum standards applicable to a factor must 
be revealed [35:22]. 


Various aspects of factors need not be explicitly 
identified provided such aspects are logically 
related to the stated factor [36]. 


Relative importance of "definitive" or descriptive 
sub-factors need not be revealed. 


Relative importance of sub-factors that are 
essential characteristics or measures of perform- 
ance shall be revealed [37]. 


will not become involved in appraising the quali- 


fications of evaluators [38]. 


in ailocations of bias, the protester has the 
buUbEaen Of proort [39]. 


The evaluation panel may be one individual or 
a team [40:8]. 


All evaluators are not required to evaluate 
revised proposals [41]. 


4. Evaluation factors may be used in the negotiation of 


fixed price contracts, particularly if the desired product is 


creativity and innovation rather than manpower [10]. 
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= Proposal evaluation should not be considered 
responsibility findings [10]. 
5. Evaluation of detailed resumes and firm commitment 
of key personnel do not create a prohibited personal services 
@entract [42]. 
- tf offerors are contractually committed to provide 
listed key personnel, letters of commitment become excess [43]. 


- Requirement that key personnel be presently 
employed with offeror is unduly restrictive [44]. 


6. It is fundamental in the award of a cost type contract 
that proposed cost be analyzed in terms of cost realism [45]. 


- Merely comparing the estimated costs proposed by 
offerors is inadequate [46]. 


= An independent cost projection should be used in 
the evaluation of the reasonableness of proposed 
GOst 4 [l7 |x 


= It is permissible to reveal the government esti- 
mate and to use it in negotiations [47]. 


= GAO has cautioned against undue reliance on the 
government cost estimate [47]. 


= Phase-in cost factors may be used only if set 
forth in the RFP [48]. 


= If regulations require the use of certain factors, 
they must be used even if not disclosed in RFP. 
Examples are transportation costs and freight 
costs. [49]. 


=_ intent to use Government property is a required 
factor if there are indications that one or some, 


but not all offerors will require the property 
[50]. 


_ Evaluated cost rather than proposed cost provides 
a more sound basis for determining the most 
advantageous proposal [51]. 
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7. It is not the function of GAO to make determinations 
as to the acceptability or relative merits of technical 
proposals [26]. 


- The TEP may consider its own experience with the 
offeror in evaluating the proposals [52!. 


- The Government may use its own reasonable esti- 
mate of manhours to measure the offeror's under- 
standing of requirements [53]. 


- In an unrestricted procurement, it 1s improper 
to evaluate small business proposals differently 
[54]. 
- Evaluation of prior experience/past performance 
1S not.impropér or discriminatory with respect 
to small business [10]. 
= The contracting activity must bear the burden of 
any difficulties resulting from a defective 
evaluation [26]. 
- If an evaluation is found to be arbitrary and 
Capricious, the low offeror may be entitled to 
proposal preparation cost [21]. 
8. Proposals that do not provide sufficient detail to 
permit technical evaluation without complete rewrite may be 
rejected as technically unacceptable [55]. 


9. Award may be based on initial proposals: [40:4] 


- if urgent delivery requirements will not permit 
discussions. 


- 1f the existence of adequate competition or 


accurate cost experience can be clearly 
demonstrated. 


- if acceptance of the most favorable initial 
proposal would result ina fair and reasonable 
price to the Government. 

10. Unless the award is to be made on a proposal as 


initially submitted, written or oral discussions must be held 


with all responsible offerors in the competitive range [40:5]. 
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ie. The 
offerors as 


deficient. 


The use of pre-determined cut-off scores to 
establish the competitive range is not in 
accordance with sound procurement practice [56]. 


Proposals may be found outside the competitive 
range on the basis of cost or technical factors 
without consideration of the other, however, a 
technically acceptable proposal may not be 
found outside the competitive range without 
consideration of cost [40:5]. 


Once a proposal is included in the competitive 
range, it cannot be excluded without an oppor- 
tunity to submit a revised proposal [40:5]. 


An unsuccessful offeror is to be advised that 
he is no longer in contention unless award is 
to be made in a few days [40:9]. 


Government must usually furnish information to 


to the areas in which their proposals are 


It is not required to discuss inadequate areas 
of a proposal to the extent technical trans- 
fusion or technical levelling would take place 
Roadie 


The offeror has a right to know that discussions 
are being conducted, Best and Final (B&F) offers 
are required, and the deadline for submission 
(40:7). 


A second B&F is permitted where justifiable 
[40:7]. 


Technical scores may be reduced after second 
evaluation (36 . 


12. Where selection officials have determined proposals 


to be technically equal, cost can become the determining fac- 


tor in award [58]. 


A determination of technical equality must be 
made. Simply awarding to the lowest qualified 
offeror fails to follow the announced evalua- 
tion scheme. [59] 


56 





- A factual explanation why proposals are essentially 
equal is required. A bare conclusionary statement 
is not sufficient. [60] 

- Whether a given point spread indicates a signifi- 
cant superiority of one proposal over another 
depends on the facts and circumstances of each 
procurement [61]. 

13. The contracting officer or source selection official 
is not bound by the recommendations of the TEP provided such 
selection is reasonable and consistent with the evaluation 
criteria [62]. 

14. Protests which are based upon alleged improprieties in 
the evaluation scheme and which are apparent prior to the 
closing date for receipt of initial proposals must be filed 
prior to the closing date for receipt of initial proposals. 
Protests alleging improprieties in the evaluation of proposals 
must be filed not later than ten days after the basis for pro- 


test becomes known or should have become know, whichever 1s 


earlier, [35:22] 


Bee PROTECTIVE FUNCTION OF SBA 

As noted carienagter PE cne) SBA has a role in the deter- 
Mination cf responsibility with respect to small business 
offerors. The SBA has statutory authority to certify the 
competency of any small business concern as to all elements 
Bemecesponsibility, including, but not limited to, capability, 
competency, capacity, credit, integrity, perseverance, and 
tenacity. The Defense Acquisition Regulation (DAR) 1-705.4 


states that the contracting officer shall accept as conclusive 
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SBA certificates of competency unless the contracting officer 
s substantial doubt as to the small business ability to 
perform. in this case the contracting officer must follow a 
review and appeal procedure set forth in DAR 1-705.4(f). 

A few contracting officers have attempted to take advan- 
tage of the fact that the use of evaluation factors to evalu- 
ate technical proposals, when required by the nature of the 
requirement, is not a responsibility determination. Evaluation 
factors have been used on requirements where highest expertise 
or best scientific approach was not required. This is indi- 
cated when the evaluaticn scheme contains only responsibility 
type factors. However, GAO has condemned the practice of 
rejecting small business proposals under the guise of relative 
assessment in order to avoid compliance with statutory require- 
Ments to submit such responsibility determinations to SBA. 


16332} 
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V. EVALUATION OF SERVICE CONTRACTORS 


A. PURPOSE 

“The heart of the evaluation and selection process is the 
evaluation factors and the weights assigned. The evaluation 
factors should consist of those elements which the Technical 
Evaluation Panel (TEP) must examine in each proposal in order 
to determine an offeror's: [64:11] 

= understanding of the service to be provided. 

_ technical, business, and management approach, 


~ potential for completing the job as specified in 
RFP. 


= probable cost based upon the offeror's approach. 


re relative qualifications and experience Pye Lud sng 
the key personnel proposed. 


= relative competitive status. 

- commitment and assumption of risk. 

The establishment of evaluation factors and their weights 
requires the exercise of judgment on a case-by-case basis 
[12:2-2]. However, it is the purpose of this chapter to pro- 
vide some general guidelines on the selection of evaluation 
factors and then some more specific quidance for service 


SemeractS in particular. 


B. GENERAL GUIDELINES 
In selecting evaluation factors for any evaluation, the 
evaluation factors should meet the following criteria. The 


Eactors should be: [65:50] 
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—_ 


- discriminating, result in different scores for 
competing proposals of various relative merit. 


- operational, can be meaningfully used in the analysis. 


- decomposable, can be simplified by breaking down into 
unidimensional parts. 


- relevant to source selection, do not collect "baseball 
SEteEoures  Ormiata on all aspects of the offeror. 


After the initial list of factors is developed, it should 
be tested as a set against the following criteria: [66:123] 


_ independent or redundant, double counting of factors 
should be avoided. 


- exhaustive or incomplete, covers all the important 
aspects of the prcposal. 


- compensating or unbalanced, high scores in one factor 
. offsets low scores in another. 


C. SPECIFIC GUIDELINES 

‘As indicated in Chapter IV, there are no restrictions on 
the kinds of evaluation factors which may be used as long as 
they are disclosed in the RFP and relate to the purpose of 


the procurement. However, a review of major weapons systems 


handbooks.and guides, and interviews with NFAS personnel has 


shown specific areas of eeMaeden common to service contract- 

ing. The specific factors generally fall into the following 

Major categories: technical; key personnel; management and 

business; experience and past performance; cost; and others. 
ime Technical 


Often contractors will be required to submit technical 


proposals that outline their scientific or engineering approach 


_to a problem statement set forth in the RFP. The SOW is very 
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broad and new, innovative ideas are encouraged. The technical 
factor may become dominant in the evaluation scheme. An 
example of how the major technical factor may be decomposed is 
as follows: [5:3-51] 

= soundness of approach. 

- understanding of the requirement. 

_ compliance with requirements in RFP. 

= unique ideas. 

2. Key Personnel 

The SOW may also be very detailed and specific with a 
fixed approach. An example would be training services with a 
fixed subject matter and an established schedule of instruction 
periods. The source selection then becomes highly dependent 
on the education, experience, and quality of the key personnel 
proposed by each offeror. When key personnel is used as an 
evaluation factor, the RFP should identify those functions or 
positions for whicn key personnel will be evaluated and require 
the submission of resumes. There are many problems related to 
the use of key personnel as a factor and the submission of 
resumes. [8] 

- The resumes submitted are too general and do 
not address the specific skiils or experience 
required for the immediate procuremt. 

- The key personnel proposed are not currently 
employed by the offeror and there may be 
uncertainty as to their availability. 

- The key personnel proposed are highly experi- 
enced and qualified but only work in adminis- 
tration and management of the firm and rarely 


provide direct contract effort [67:21]. 
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- Certain individuals appear on multiple 
proposals. 


- Key personnel may depart shortly after award 
of the contract or be used on other contracts. 


- There may be a mismatch between the qualifi- 
cations of key personnel and the wages reported 
in the cost proposal. 

As stated in Chapter IV, the Government cannot require 
that key personnel be currently employed by the offeror be- 
cause it unduly restricts competition[44]. Many NFAS actiti- 
ties require the offerors to submit letters of commitment 
from key personnel with their proposal. Several activities 
have developed special contract clauses that require that key 
personnel must be replaced by substantially equal personnel. 
Failure to do so then becomes a grounds for Termination for 
Default [8]. The special contract clause for key personnel 
used by NSC Oakland is provided as figure 9. 

Many of the above problems common in the use of 
resumes can be traced to inadequate solicitations. The RFP 
must clearly identify the key personnel positions and provide 
explicit direction on the content and form of the resumes. 

If appropriate, the RFP should require the contractor to pro- 
yide personal references and contact points in order for the 

TEP to validate an individual's performance on a past work 
effort. Often in the wording of resumes, the contractor may 
Overstate or conceal the relatively minor role of an individual. 
Also, the contractor may fail to state that the customer was 


dissatisfied with the performance of the individual [67:21]. 
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KEY PERSONNEL REQUIREMENTS CLAUSE 


(a) Certain skilled experienced professional and/or technical 
personnel are essential for successful contractor accom- 
plishment of the work to be performed under this contract. 
These are defined as "Key Personnel" and are those per- 
sons whose resumes were submitted for evaluation of the 
proposal. 


The contractor agrees that such personnel shall not be 
removed from the contract work or replaced without com- 
pliance with paragraphs (b) and (c) hereof. 


(b) If one or more of the key personnel for whatever reason 
becomes, or is expected to become, unavailable for work 
under this contract for a continuous period exceeding 30 
work days, or is expected to devote substantially less 
effort to the work than indicated in the proposal or 
initially anticipated, the contractor shall immediately 
notify the Contracting Officer and shall, subject to 
the concurrence of the Contracting Officer or his auth- 
orized representative, promptly replace such personnel 
with personnel of at least substantially equal ability 
amcegqualifications. 


(c) All requests for approval of substitutions hereunder 
must be in writing and provide a detailed.explanation of 
the circumstances necessitating the proposed substitu- 
tions. They must contain a complete resume for the pro- 
posed substitute, and any other information requested 
by the Contracting Officer or needed by him to approve 
or disapprove the proposed substitution. The Contracting 
Officer or his authorized representative will evaluate 
such requests and promptly notify the contractor of his 
approval or disapproval thereof in writing. 


(d) If the Contracting Officer determines that suitable and 
timely replacement of key personnel who have been 
reassigned, terminated or have otherwise become unavail- 
able for the contract work is not reasonably forthcoming 
or that the resultant reduction of productive effort 
would be so substantial as to impair the successful 
completion of the contract or the service order, the 
contract may be terminated by the Contracting Officer 
for default or for the convenience of the Government, 
Gemappropriate, or, at the discretion of the Contracting 
Officer if he finds the contractor at fault for the 
condition, the contract price or fixed fee may be 
equitably adjusted downward to compensate the Government 
for any resultant delay, loss or damage. 


USE: In RFP"*s and contracts where having skilled and experi- 
enced technical or professional personnel constitutes an 


important factor in the evaluation of technical proposals 
for award. 


Figure 9 
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If the TEP is not satisfied with the resume or an individual, 
the oral or written negotiations required to be held with 
those offerors in the competitive range should be used to 
gather additional data [12:2-5]. If the TEP simply is not 
satisfied with the individual, this should be pointed out as 
a deficiency in order to allow the contractor the opportunity 
to propose a more satisfactory individual. 

3. Management and Business 

An example of a further breakdown of the major manage- 
ment and business factor is as follows: 

= Management and business. 

- management plan. 
- planning. 
a OLgGani Zand 
reo iietn oie toric. 
- Corporate resources. 
- necessary support equipment. 
- Manpower skill mixes and number. 
- necessary support facilities. 

Management science has clearly identified planning, 
Organizing, and controlling as the three basic functions of 
Management [68:92]. The management plan describes the 
offeror's approach for efficiently managing the work and 
effectively integrating these functions. It normally includes: 


[12;2-4] 


- the proposed organization including internal 
Operations and lines of authority. 
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- external relationships and interfaces with 
the Government. 


~ major subcontractors. 

= necessary performance schedules. 

The management plan can become an important discrimi- 
nator in contracting for services for the operation of 
Government owned facilities and systems. Often the majority 
of the operational personnel remain while the management 
meeups Change from contract to contract. Therefore, the pri- 
mary function of the contractor is to provide management 
Seats, [11] 

A contractor may also locate, staff and equip a 
Facility physically close to a major Government activity in 
Order to provide support services. Examples are R&D services, 
engineering and technical services, and ADPE services. The 
discriminating evaluation factors could then be related to 
Superior computer capability, superior evaluation and test 
equipment, or superior facility features [{12:2-6]. In this 
evaluation area, great care should be taken to separate quali- 
fication factors from evaluation factors. For example, a pro- 
curement may require that a contractor provide a service that 
necessitates the use of a computer compatible with Government 
computers. The computer compatibility requirement should be 
meemomre aS a qualification factor. Evaluation factors for 
other computer compatibilities may or may not be used in the 
evaluation plan. If the qualification factors are well under- 
Stood, contractors lacking the required personnel, equipment, 
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or facilities are not likely to incur the expense of proposal 
preparation [16:3-15]. 

It may be necessary to evaluate personnel as a corpor- 
ate resource in place of or in addition to key personnel. An 
example would be senior engineers, engineers, draftsmen, and 
technicians required to provide engineering support services. 
A proper skill mix would provide adequate services without 
paying for over qualified personnel. The proper numbers of 
personnel would relate to the schedule being offered. 

In the overall area of corporate resources, if an 
offeror does not possess adequate resources himself, he may be 
able to demonstrate the ability to acquire them through sub- 
contracts or otherwise [12:2-6]. 

4. Experience and Past Performance 

Past performance and experience are often used inter- 
Changeably but there is a difference in their meaning. Past 
performance is how well an offeror did on earlier work and 
can be a very significant indicator of how well he can be 
expected to perform on the proposed contract. Many organiza- 
tions exhibit characteristics that persist over time. Examples 
are resiliency in the face of trouble, resourcefulness, manage- 
Ment determination, skill in developing key personnel, etc. 

Experience is the accomplishment of work by an offeror 
which is comparable or related to the work or effort required. 


Experience includes technical, cost, schedule, and management 


elements. 
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This evaluation factor should be limited to the over- 
all corporate or offeror experience and past performance and 
not the past performance and experience of individuals. 
Individuals should be evaluated under key personnel. Informa- 
tion and data should be collected in addition to the proposal 
submitted by the offeror. Admittedly, the assessments to be 
made regarding experience and particularly past performance 
may be very difficult. Often the data will not be clear 
because of the difficulty in measuring recorded performance. 

In the event a substantially unfavorable response is received 
from another Government activity, the offeror should be given 
an opportunity durinu discussions to submit written comments 
regarding the unfavorable response. [12:2-8] 

It may be prudent to include contractor potential 
With experience and past performance. New corporations 
dissolve and reform at a very rapid pace and it is not uncommon 
to be faced with the evaluation of a highly qualified contrac- 
tor with no corporate history. Including potential provides 
the TEP with considerable flexibility. [8] 

5. Cost/Price 

Cost or price is a required factor that must be included 
in the evaluation plan and relative importance revealed in the 
RFP. However, there is a disagreement as to whether cost/price 
Should be scored in the same manner as other factors or whether 
cost/price should be evaluated in a narrative manner. 

From a theoretical view, there is no difference be- 


Cween cost/price and any other factor in the evaluation plan. 
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It must be converted to the same utility scale in order to 
evaluate the trade-offs between factors [24]. Most NFAS 
activities score cost/price using a variety of scoring 
methods [8]. 

The GAO prefers that cost/price not be scored. They 
prefer that after a complete technical evaluation, the con- 
tracting officer or source selection official should prepare 
a narrative justification for the expenditure of funds over 
and above that proposed by the lowest qualified offeror in 
the competitive range. The rationale behind this position is 
that all factors other than cost/price are judgmental and 
therefore subjective. On the other hand, cost/price is a 
totally objective value and scoring distorts an already clear 
value. Most major weapons systems activities and NASA do not 
score cost. Because this issue is not addressed in statute, 
GAO accepts both methods. [69] 

By whatever method cost/price is to be evaluated, a 
clear determination must be made as to what elements are to 
be included in the cost/price evaluation. Certain elements 
are required by regulations and must be considered in the 
evaluation even if they are not listed in the RFP. Examples 
are fright costs, transportation cost and Government Furnished 
Material (GFM). [49] 

Options may be included in a service contract if there 
is an anticipated need beyond the initial contract period. 
This is in recognition of the Government's need for continuity 
Of operations and to preclude cost associated with disrupted 
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support. Normally evaluations will be made exclusive of the 
option period. DAR 7-2003.11 provide special clauses for the 
evaluation of options and to discourage the submission of 
unbalanced proposals. An unbalanced proposal is one which is 
based on prices significantly less than cost for one period 
and prices significantly more than cost for another period. 

An unbalanced proposal may be rejected as non-responsive. 

The option periods may be considered in fixed price 
type contracts if realistic competition is impracticable after 
the initial contract is awarded. This may be a result of sub- 
stantial startup or phase-in costs or superior technical 
ability resulting from performance of the initial contract. 
Options may also be considered if there is a known requirement 
Beyond the initial period and funding is currently unavailable. 
A reasonable certainty must exist that funds will be available 
to exercise the option. [70:1-1500] 

Start-up or phase-in cost to the Government may be 
assigned to all proposals other than the incumbent contractor's 
proposal if specifically set forth in the RFP [48]. This 
naturally favors the incumbent and the procurement activity 
should expect to defend the amount, relevance and need of the 


costs assigned. 


D. TESTING THE EVALUATION FACTORS 
The evaluation factors may be tested for appropriateness 
by using different perspectives. From the potential contrac- 


tor's perspective, the evaluation factors must make sense in 
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terms of the SOW requirements. From the TEP perspective, there 
should be a match between the evaluation factors and the skills 
and expertise of the TEP. The TEP must be able to develop 
documentation to justify the differences between acceptable 
offerors (lowest acceptable to highest acceptable). From the 
contracting officer's perspective the evaluation factors must 
not violate the rules summarized in Chapter IV. The evaluation 
factors must also be viewed from the perspective of future 
unsuccessful offerors. Although not violating any particular 
regulation or rule, the evaluation factors may unnecessarily 


invite protest. 


E. THE FINAL EVALUATION PLAN 

The RFP is only required to reveal the major evaluation 
factors and their relative importance. However, a final and 
complete evaluation plan should be developed early to avoid 
accusations that the evaluation plan favors certain offers. 
This may be prior to the release of the RFP, but certainly 
prior to the opening of any proposals. The evaluation plan 
Should contain the weights, scoring/rating system and the 
basis for recommendation for award. Some complex normalization 
procedures have been used in an attempt to increase accuracy. 
For example, the measure of standard deviations from the 
average technical score as a factor to apply to raw scores. 
Any reasonable system is acceptable as long as it does not give 
unreasonable results. However, a system such as MAUM is sup- 
ported by sound decision theory and extensive field research. 
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The issue of scoring cost is very sensitive and should be 
settled by making it a part of the evaluation plan. [In the 
NFAS activities interviewed, the cost proposal is removed from 
Mme otferor’s proposal prior to the initial technical evalua- 
tion by the TEP. The cost proposal is evaluated by the con- 
tracting personnel and the cost proposal is later provided to 
the TEP to check compatibility and reasonableness with the 
technical proposal. The cost proposal is scored by the con- 
tracting personnel often working closely with the TEP [81]. 
Local activities may develop special methods for assigning 
points to cost. Two such methods are the Twice-Low Method and 
the Greatest Value Method. 

Twice-Low Method: NSC, Oakland has developed the Twice- 
Low Method to assign points for cost factors. The method is 
best explained by an example. If a maximum of 20 points was 
possible, the lowest acceptable offer would receive the full 
20 points. The cost proposal of the lowest offer is then 
doubled and any offeror with a cost proposal greater than that 
amount would receive a score of zero. Any cost proposal within 
this range is scored on a straight linear scale. Therefore, a 
cost proposal halfway in between would receive a score of 10. 
71) . 

Greatest Value Method: NRCO, Long Beach has developed 
the Greatest Value Method. In this method the cost proposal 
of the lowest offeror is divided by the cost proposal of the 


offeror being evaluated and then multiplied by the maximum 
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points possible. This also results in the lowest offeror 
receiving the maximum score possible. Therefore, if low 
acceptable offeror A proposed $150,000 and next offeror B 
proposed $200,000, of a possible 30 points, offeror A would 
receive 30 and offeror B would receive 22.5. 

Both activities recognize that these methods are only 
tools and that each point assignment must be supported by 
written narrative judgment. Variations from using these 
mathematical formulas are allowed but must also be properly 
Supported in writing. 

The TEP and contracting personnei should clearly 
establish the basis of award recommendation. Normally, the 
offeror with the highest combined technical and cost score is 
recommended for award. However, award may be on a different 
basis. For example, award may also be made to the acceptable 
offeror in the competitive range with the lowest cost/price 
proposal. The basis for such an award must be set forth in 


Mie RFP. [59] 


F. PROPOSAL PREPARATION GUIDANCE 

After careful preparation of the SOW and SSP, the only 
major component remaining prior to release of the RFP is the 
Proposal Preparation Guidance (PPG). Again, it is important 
that the PPG be consistent with the SOW and evaluation factors 
Oemene REP, Explicit instructions in format as well as 


implicit guidance contained in the organization and layout of 
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the RFP indicate to the offeror what the Government would like 
to see in the proposal. 

Proposals are expensive and time consuming for the offeror 
to prepare and for the Government to evaluate. While proposals 
are expected to be comprehensive, excess volume, detail and 
brochuremanship do not serve either party. Misunderstandings 
and needless expenses are avoided if the RFP clearly estab- 
lishes what is important and how to prepare proposals. To 
control size, the Government must exercise restraint in the 
listing of requirements and in the detail demanded. The esti- 
mated value of the contract should be used to strike a balance 
between what the Government must have and what would be nice to 
have. 

The RFP should direct the offeror how to organize and 
arrange the proposals. This will not only facilitate the evalu- 
ation tasks of the TEP but will also facilitate proposal prepa- 
ration. Great care should be taken in drafting these 
instructions in order that desirable innovations are not pre- 
Cluded [16:3-16]. The solicitation should include only 
necessary formats and concepts rather than an elaborate instruc- 
tion for preparing proposals. The offerors should be directed 
to submit a sufficient number of copies of their proposal in 
Order that each TEP member shall have his own copy. 

The RFP should have a specific list all in one place of 
the data needed in the proposal. The list should be compre- 


hensive and not depend on the offeror to fill in the blanks. 
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The instructions should motivate the contractor to submit a 
"Pirst and Final" proposal and not to depend on negotiations 


to correct any deficiencies [7]. 


G. PREPROPOSAL CONFERENCE 

Since preproposal conferences are expensive and time con- 
suming to host and attendees, they should be used sparingly. 

A preproposal conference may be necessary in order to provide 
information that cannot be provided in the RFP or to clarify 
nebulous points. Examples are tours of Government facilities 
to be operated or formal presentations to further explain 
complex requirements. [5:4-20] 

When the decision has been made to hold a preproposal 
conference, the procurement personnel make the necessary 
arrangements. Adequate notice shall be given to prospective 
offerors in order that they may submit questions and make 
arrangements to attend. The exact nature and scope of the 
conference should be explicitly stated. 

The conference itself should be conducted by the contract- 
ing officer or a contract specialist. The TEP will assist and 
Should be briefed on what can and cannot be discussed. Care 
Should he taken to place emphasis consistent with the evalua- 
tion factors and SOW. All offerors should be provided identi- 
cal information and advised that unless the RFP is amended in 
writing, it remains unchanged. A complete record shall be 
made of the preproposal conference and made a part of the con- 
mBract file. 
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VI. CONDUCTING THE EVALUATION 


A. PURPOSE 

The purpose of this chapter is to list and describe the 
essential steps of the evaluation process. It is at this 
point that the extra time and effort invested in planning pays 
off. The primary function of the evaluation process 1s to pro- 
vide a sound recommendation to the contracting officer in 
order that he may make an informed and objective source selec- 
tion. This may consist of a single evaluation, but more often 
additional data gathering, negotiations, revised proposals 
and a second evaluation are required. This makes the process 
a combination of factfinding, reporting and application of 


professional judgment. [16:3-24] 


B. TEP FAMILIARIZATION 

Just prior to the receipt of proposals from the contract- 
ing officer each Technical Evaluation Panel (TEP) member 
Should review the Request for Proposal (RFP) to clearly 
establish how the Statement of Work (SOW) was presented and 
exactly how the evaluation criteria relate to this require- 
ment. Several months may have passed and solicitation amend- 
ments may have been issued. Each evaluator must understand 
that the evaluation is to be accomplished in accordance with 
the RFP as written. What each evaluator thinks it should say 


Cannot be allowed to enter the evaluation process. 
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A meeting of the TEP should be held to review and update 
the SSP. The TEP chief should chair the meeting. The primary 
purpose of the meeting is to cover all the administrative 
details of conducting the evaluation and to establish a 
schedule for evaluation completion [5:4-22]. Ideally, TEP 
members should be divested of all other duties and provided 
a separate working area to conduct the evaluations [16:2-7]. 
However, most NFAS activity customers are unable to do this 
and each TEP member must balance TEP duties with regular 
office duties. These problems must be confronted and solutions 
negotiated because the longer the evaluation drags out the 
greater the chance that either the cost proposals will expire 
and therefore need revision or the confidentiality of a pro- 


posal will be violated [11]. 


C. RECEIPT AND BREAKOUT OF PROPOSALS 

DAR does not have a detailed procedure for the receipt, 
storage, and opening of proposals received in a negotiated 
procurement. However, the NFAS activities interviewed have 
well-developed procedures. All proposals, revised proposals, 
letters, telegrams and other correspondence are received by 
the contracting office. The original documents are made a 
permanent part of the contract file and only copies are dis- 
tributed to the TEP. The cost proposals are usually separated 
from the technical proposal and provided to the TEP only after 
the initial technical evaluation is completed. A few activi- 


ties have gone so far as to mask the identity of each offeror 
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in order to increase objectivity. The proposals are forwarded 
Memene technical activity by U. S. mail with a cover letter 
that includes the appropriate cautions {ll}. 

Technical activities have a less structured proposal con- 
trol procedure. Classified proposals are controlled by the 
same security procedure as all classified documents. Due to 
the collateral nature of the TEP position, unclassified evalu- 
ations are often conducted at home during the evening or on 
weekends. Security and efficiency can be improved if each 
evaluator has a complete set of proposals. Each evaluator can 
then be individually accountable for a particular set of pro- 
posals. If for some reason proposals must be routed, delivery 


is by hand and not by guardmail. [11] 


D. INITIAL EVALUATION 

1. Qualification Factors 

The initial evaluation start with a review of each 

proposal using the qualification factors. This is a very 
mechanical process with a strict set of rules. The objective 
facts make a contractor either qualified or not qualified and 
professional judgment is seldom involved. Proposals that do 
hot meet the qualification factors must be removed from compe- 
tition or the procurement activity is open to protest from 
competitors. 

2. Grossly Deficient Proposals 

During this initial review, any proposal that is so grossly 


deficient that it does not provide a reasonable basis on which 


ay 





to conduct an initial evaluation without a total rewrite may 
be rejected as technically unacceptable [55]. Surprisingly, 
incumbent offerors may assume that the Government is familiar 
with their capabilities and decide not to incur the expense 
of a detailed proposal. Evaluation panels have rejected pro- 
posals from incumbents as grossly deficient and eliminated 
them from further consideration. Grossly deficient proposals 
are from contractors that do not have a basic understanding 
of Government contracting in general or of the anticipated 
work effort in particular. It is not the function of the TEP 
to provide basic lessons in the proper preparation of proposals 
[15]. A note of caution must be taken to ensure that the 
problem is with the offeror and not the result of a grossly 


deficient RFP [9]. 


E. IN-DEPTH EVALUATION 
1. Considerations During Evaluation 
There is a lot of gamesmanship and brochuremanship 
used in the preparation of proposals by offerors. Some points 
to consider when performing a technical evaluation are: 
[6;IV-7] 


_ avoid "reading into" or "reading out of" thereby 
overly interpreting what is written. 


— avoid the influence of first impressions. 
= look for ambiguities and inconsistencies. 


- recognize the use of "catch phrases" and "buzz 
words." 
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- recognize the difference between technical 
substance and glossy presentation format. 


- recognize the existence of flattery by an 
offeror. 


Some of the games that offerors may play are: 
mei 27 | 

= on a cost type contract an offeror may overstate 
his future business base in order to substantiate 
a relatively low overhead rate. 

= on a cost type contract the offeror may optimisti- 
cally underestimate the direct labor hours for 
a function such as engineering design in order 
to propose the lowest estimate cost. 

= an offeror may hold a lower Best and Final figure 
for use at the last moment in an effort to show 
a competitive attitude or in hopes of obtaining 
last minute knowledge of what the competition 
is doing. 

= an offeror may hire a competitor's employee in 
an effort to validate or enhance his knowledge of 
the competition. 


- an offeror may attempt to "buy-in" with hopes of 
Making profits on changes. 


2. Errors, Omissions and Deficiencies 

While conducting the evaluation tne TEP may encounter 
errors, omissions and deficiencies, some of which are super- 
ficial and easily corrected while others are more serious. 
Errors are a mistake in calculation or measurement or 4 minor 
misconception. An omission is simply a failure to respond to 
a particular requirement in the RFP. 

A deficiency may be any part of an offeror's proposal 


Which; [16;3-24] 
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- fails to meet the minimum requirements of a 
standard set forth in the RFP. 


- proposes an approach that has unacceptable 
Pisk, 


- describes an approach taken by the offeror 
which yields undesirable performance. 


3. The Evaluation 

ho stated in Chapter fl, after all the efforts to 
develop a comprehensive and effective evaluation structure, 
the individual evaluator is still faced with the problem of 
exactly how to evaluate the proposals. The problem centers 
on assigning scores in such a manner as to maintain consist- 
ency for each evaluator and comparability between the evalua- 
tors. Evaluators must not make the mistake of rating the 
proposals in relation to each other. Individual evaluators 
tend to choose a single proposal as a base and rate other 
proposals in relation to the base proposal. Other evaluators 
May use a mixture of proposals as a base or not have a base. 
Because the individual evaluators are not evaluating to the 
same base, their findings are less consistent and comparable. 

A well written RFP should have clearly established 
standards and serve as a single basis against which to compare 
proposals. The oversimplified procedure is to take each eval- 
uation factor and compare the proposal against the standard in 
the RFP. Using his judgment and expertise, the variance is 
determined by the evaluator, described in narrative form, and 
represented in numerical form. [14:27] 

Care must also be taken to ensure that the evaluation 


does not center just on errors, omissions and deficiencies. 
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Positive factors that are above the standards can offset 
other negative factors. Therefore, it 1s necessary to 
consider not only what is positive or negative, but also the 
degree to which it is good or bad. In the case of negative 
meteors, the difficulty of correcting the factor must be 
fully assessed. The impact of a negative factor may be so 
great as to make the entire proposal unacceptable or so minor 
as to be insignificant. In the case of a positive element, 
the evaluator must determine not only the excess above the 
standard, but also whether the excess represents a useful 
value to the Government. [14:29] 
4, Requests for Additional Information 

The TEP has a natural desire to start the negotiations 
before they get done with the initial evaluation [ll]. TEP 
members are frustrated by what appear to be simple omissions 
and errors in the proposals. Regulations permit the TEP to 
obtain clarifications from the offeror during the evaluation 
process. The evaluators prefer to request these clarifica- 
tions as they arise claiming the evaluation cannot proceed 
until they are clarified. This can result in a continuous 
stream of questions from the TEP and answers from the offerors 
flowing through the contracting office. The combined result 
is the conversion of clarifications into negotiations and 
extensive delays in completion of the initial evaluation [7]. 

The TEP must at least exercise restraint and NASA has 


gone so far as to forbid any questions until negotiations are 
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formally initiated. NASA evaluators are directed to complete 
evaluations based on the most reasonable assumptions they can 
make regarding the proposal areas involved. Their assumptions 
and the additional information are the subject of oral/written 
discussions with those firms determined to be in the competi- 
tive range. This has the added advantage of avoiding unneces- 
Sary communication with offerors outside the competitive 
range. [20:3-6] 

5. Cost/Price Considerations 

a. Cost/Price Analysis 

Regardless of the type of contract, the relative 
value of cost/price, or whether cost/price is scored or not 
scored, a complete analysis of the cost/price proposal is 
mequired. It is at this point that the differences in 
resources available to the major weapons systems activities 
and the NFAS activities become significant. The major weapons 
Systems activities often have professional cost/price analysts 
that provide support to the contracting officer and TEP. 

Only a few cost/price analysts are currently assigned to NFAS 
activities. At the NFAS activities interviewed, the cost/price 
analyst function was performed by contract specialists with 
primary support from the Defense Contract Audit Agency DCAA) 

18] .. 

A DCAA audit is very time consuming with sixty day 
deliveries being common at the NFAS activities interviewed. 
The NFAS activities recognize that much of this delay is 
Caused hy excessive backlog at DCAA and therefore take measures 
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to limit the number of DCAA audit requests. The primary 
method is to delay the request for DCAA audit until after 
the TEP has completed a preliminary evaluation using qualifi- 
cation factors and searching for grossly deficient proposals. 
This eliminates the need for DCAA to conduct an unnecessary 
audit on an offeror that does not have a chance for award. 
Another method, used primarily on smaller service contracts, 
is to limit the DCAA audit to a rate verification conducted 
through the resident DCAA Liaison Office. A rate verification 
can be completed in only days and is quite sufficient if the 
offeror has a substantial volume of Government business, 
material is not involved and rates have not changed. However, 
1f a significant problem in rates is uncovered, a complete 
audit is necessary [8]. The last method available to reduce 
the number of DCAA audit requests is to wait until after a 
complete technical evaluation. This will eliminate those 
offerors that are not technical acceptable. The goal of the 
NFAS activities is to hold the number of DCAA audit requests 
per evaluation to no more than three, if possible [4]. 
Another major aid in the cost/price analysis is 
the independent cost estimate (ICE) developed as part of the 
Por. Freparation of an ICE is discussed in Chapter II. A 
well prepared ICE adds another dimension to the cost/price 
analysis and can be a critical aid if action must be taken 
without the benefit of a DCAA audit. Unfortunately, the 


ability or resources of most NFAS activities and their 
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customers in ICE preparation is limited. Furthermore, an ICE 
may be inaccurate due to the performance uncertainties inherent 
particulariy in cost type contracts. As a result, GAO has 
cautioned against undue reliance on the ICE. [47] 

Another cost/price analysis method is to compare 
the cost proposals of all offerors. For fixed price require- 
ments that do not have evaluation factors, this may be the 
entire basis for the contracting officer's determination that 
the price is fair and reasonable. However, cost/price analysis 
has a different objective when used in conjunction with require- 
ments. that have evaluation factors. That objective is source 
eee -eeron and a fair and reasonable price determination. The 
GAO has found that when the cost/price analysis consists of 
merely comparing the estimated cost proposals submitted by 
offerors, rational support for the source selection does not 
exist due to inadequate analysis and assessment. [46] 

Cost/price analysis, as required by DAR 3.807.2, 
may be accomplished in various ways including DCAA audit, 
comparison against an ICE, and comparison of proposals sub- 
mitted. GAO encourages the use of all information available 
at the time of evaluation. [54] 

bh. Cost Realism/Probable Cost 

It is fundamental in the award of a cost type 
contract that proposed cost be analyzed in terms of cost 
realism [45]. At the NFAS activities interviewed, cost 
realism was considered after completion of the technical evalu- 


ation by the TEP and completion of the cost analysis by the 
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contracting personnel. By comparing the technical proposal 
mith the cost proposal, the TEP and contracting personnel 
determine: (ira 5 20) 
- that the offeror's cost estimate is compatible 
with the technical and management effort 


proposed. 


- that all costs have been included and that 
estimated costs are relatively valid. 


- that the costs are in consonance with the 
work statements in the RFP. 


Significant inconsistencies between the technical 
proposal and the cost proposal can be considered in two ways. 
The inconsistencies may raise a fundamental issue concerning 
the offeror's understanding of the nature and scope of the 
work required. This may result in appropriate areas of the 
technical evaluation being re-scored. The evaluators may be 
able to determine a more probable cost than that proposed by 
the offeror [6:IV-6]. It is then appropriate to use the 
probable cost to score or evaluate the cost proposal [58]. 
Any Significant inconsistency will be the subject of negotia- 
tions and the burden of proof as to the credibility of the 
proposed cost rests with the offerors. 

Cost realism can also be an important factor in 
fixed price contracting and a defense against the "buy-in." 
There is nothing illegal about a "buy-in" and it is not an 
acceptable hasis for rejecting a proposal [9]. However, lack 
of cost realism can affect the technical evaluation in much 


the same was as in cost type contracting. Also, if cost is 
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being separately scored, a cost proposal can be scored lower 
for lack of cost realism. A few individuals interviewed 
cautioned against being too heavy handed and attempting to 
declare an unusually low offeror as unacceptable based solely 
on lack of cost realism. Much less justification is required 
for reducing an offeror's score than for removing the offeror 
from further consideration. In addition, a reduced score is 


much less likely to provoke a protest. [8] 


F. RESULTS OF INITIAL EVALUATION 
l. Award on Initial Proposals 

Part of the boilerplate of any contract is a statement 
that award may be made on the basis of initial proposals. 
There are two basic situations that may occur where award on 
the hasis of initial proposals may be appropriate. The urgency 
of the delivery requirements may not permit negotiations or 
negotiations simply are not necessary. The Government must 
demonstrate that adequate competition does exist and that 
acceptance of the most favorable initial proposal, without 
further discussion, would result in a fair and reasonable cost/ 
price to the Government. This is proper only if the proposal 
is accepted exactly as submitted without any negotiations 
whatsoever. 

The decision to award on the basis of an initial pro- 
posal is within the discretion of the Government. Although 
the above conditions may be met, the Government may choose to 


negotiate for a variety of reasons. The regulations do not 
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require an award without discussion, they only permit such an 
award [40:4]. Individuals interviewed stated that award on 
the basis of initial proposals is not common. However, they 
considered it important that the offeror fully realize that 
the possibility of award on the basis of initial proposal does 
erst, This forces the offeror to put more effort into the 
initial submission and not depend on the negotiation process 
to provide clues as to the Government's true desires. Also, 
it forces the offeror to submit a more realistic cost/price 
proposal and not hold back for the Best and Final (B&F). The 
NFAS activities interviewed routinely conduct negotiations and 
request a B&F and do not actively consider award on the basis 
Seeinitial proposals [ll]. 

feeb irste Technical Evaluation Report (TER) 

The primary objective of the TER is to clearly and 
ohjectively state the results of the proposal evaluation by 
the TEP. This documentation is the only official record 
Showing the logic and rationale used by the TEP members when 
reaching their conclusion. A brief discussion of significant 
weaknesses and strengths is to be provided to support the 
score assigned under each evaluation factor. It is important 
that a substantive TER be created in order to: [534-25] 

- provide the contracting officer with the necessary 

information in order to notify unsuccessful 
OEteCrors . 


_ serve to refresh the memories of TEP members 
during later debriefings with rejected offerors. 
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- to become the official documentation should 
the award be questions. 


- Support the competitive range determination. 

- aid in the development of negotiation objectives. 

Naturally, if :the TEP is recommending an award based 
on initial proposals, there is no need for the development 
of negotiation objectives or a competitive range determination. 
However, it would be prudent to document the reasons why 
further negotiations are not required or in the best interests 
of the Government. Contractors cften challenge either form- 
ally or informally, an award based on initial proposals [ll]. 

While there is no standard format for a TER, certain 
generalizations can be made. [5:4-26] 


™. 


= Findings must be supported by rational judgment 
or technical data. 


- Supporting data, findings, technical analysis 
and judgment statements are to be presented so 
people with nontechnical backgrounds can under- 
stand as much as possible. 


~ The selection of words should be consistent 
with the findinds of the TEP. An outstanding 
offer should be described in superlative terms 
rather than mediocre terms. 


_ The TER is a summary and great bulk is not 
required or desired. All supporting data is to 
be retained as part of the contract file. 

_ A primary function of the TER is to categorize 
each proposal as either acceptable, unacceptable 
but capable of being made acceptable without 
major revision, or unacceptable. 

This is in preparation for the competitive range 

determination to be made by the contracting officer. Offerors 


with unacceptable proposals are notified by the contracting 
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officer and are no longer considered. Each of the remaining 
offerors are then ranked and it is important that the relative 
rank be clearly stated. Differences among acceptable proposals 
are critical factors in the subsequent decisions related to 
selection of sources for negotiations [5:4-24]. The concepts 
of “significant difference" and "essentially technically equal” 
will be discussed later in this chapter. 

The TER should address the areas of the remaining pro- 
posals which need to be strengthened to correct weaknesses or 
areas which should be eliminated or reduced to correct excesses. 
Their recommended changes should be accompanied by appropriate 
rationale to be used in developing the negotiation objectives. 

After the TER is written, reviewed, and signed, it is 
forwarded to the contracting officer. The contracting officer 
is responsible for reviewing the accuracy and completeness of 
the documentation. This review provides a check on the 
reasoning behind the justification of the proposal ratings and 
a check that the evaluation was conducted in accordance with 
the evaluation factors in the RFP. Any nonconforming docu- 
Mentation Shall be returned to the TEP for completeness, cor- 
rection or other remedial action. [20:13-7] 

3. Competitive Range Determination 

The contracting officer is also responsible for 
selecting those sources with whom negotiations will be con- 
ducted. A competitive range decision is based on cost/price, 


technical and other salient factors and includes all proposals 
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that have a reasonable chance of being selected for award. 

A competitive range means that two or more proposals are accept- 
able for negotiation after evaluation. If there is any doubt 

as to whether a proposal is in the competitive range, it must 

be resolved by including it. A summary of the rules pertaining 
to competitive range determinations is listed in Chapter IV. 
Those sources found outside the competitive range are labeled 


unacceptable and are so notified. [5:4-26] 


G. CONDUCTING THE NEGOTIATIONS 
1. Development of Negotiation Objectives 

The Government is required to conduct negotiations with 
all offerors in the competitive range and these negotiations 
must be meaningful. Normally, negotiations will consist of 
pointing out those areas in which the proposal is deficient 
157]. The common problems encountered in this area are tech- 
nical leveling and technical transfusion. Technical leveling 
is the improper use of negotiations resulting in multiple 
offerors with approximately the same technical score. Techni- 
cal transfusion is the improper transfer of innovative ideas 
and approaches from one proposal to another. Obviously, 
technical transfusion is one of the ways to bring about techni- 
cal leveling. Another common way is to concentrate negotia- 
tion efforts on proposals that are currently unacceptable but 
capable of being made acceptable. For example, at NRCO, Long 


Beach a recent protest centered on an offeror that was able to 


90 





improve his score from seventy-five to eighty-eight due 
largely to deficiencies pointed out during negotiations. An 
offeror with an original score of eighty-five was unable to 
improve his proposal and protested because the Government 
failed to point out deficiencies known in his already accept- 
able, but not perfect proposal [ll]. The way to avoid techni- 
cal leveling is to point out all significant deficiencies in 
all proposals. What is significant must be determined for 
each individual case but generally means any deficiency that 
may affect the source selection results. 

Great care should be taken by all Government personnel 
to prevent technical transfusion. The evaluation of each 
proposal against the standards set forth in the RFP instead of 
against each other is a great aid in this effort. Each defici- 
ency is expressed in terms of how it fails to meet a standard 
or is in great excess of a standard. Deficiencies are not 
expressed in terms of how another proposal is superior. 
Evaluation to standards assists the conversion of deficiencies 
to negotiation objectives without technical transfusion. If 
it is not possible to discuss an inadequate area without tech- 
nical transfusion, then negotiations. concerning that area can- 
Bee pe held. [57] 

Another consideration is later requests for informa- 
tion under the Freedom of Information Act (FOIA). TEP members 
should write all proposal evaluations recogniZing that after 


award the evaluation report may be provided to the offeror 
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that submitted the proposal. This may result in technical 
transfusion if the evaluation report contains references to 
other proposals. 

The selection of negotiation objectives will not be 
Meeticult if an orderly method of identification, description 
and recording of errors, omissions and deficiencies has been 
followed during the evaluation process. The negotiation 
objective may also address scheduling, cost/price considera- 
tions, incentive arrangements or special provisions to be 
incorporated into the contractual document. The contracting 
officer is ultimately responsible for selecting the negotia- 
tion objectives. [16:3-26] 

2. Strategies and Tactics 

Many individuals have placed a lot of weight on 
negotiation strategies and tactics but adequate preparations 
outweights any maneuvers used during negotiations. The con- 
tracting officer will represent the Government in any negotia- 
tion and will determine the strategies and tactics to be used. 
Fach contracting officer has an individual style for conducting 
negotiation. Therefore, it is important that any TEP member 
asked to participate fully understands how the contracting 
officer intends to run the negotiations. In particular, the 
TEP member must understand whether he has a free rein to talk 
or if he must wait for approval from the contracting officer 
before engaging in discussions. Much will depend on the 


atmosphere of the negotiations. [5:4-29] 
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Conducting negotiations with several offerors ina 
competitive procurement action is a very complex process with 
severe restrictions brought about by the requirements for 
equal treatment of offerors, the prevention of technical trans- 
fusion and the prohibitions against auction techniques. This 
requires a more structured approach to the preparation for 
negotiations and a more disciplined participation by TEP mem- 
bers than would be used in sole source negotiations. The cost 
and fee area, which is the primary area of concern ina sole 
source procurement, becomes secondary to the technical nego- 
tiations. This is largely due to the Government assumption 
that cost and fee are being held down by the pressures of compe- 
tition and the greater weight assigned to technical factors. 
[5:4-30] 

3. Best and Final (B&F) 

There are two somewhat unrelated goals of the negotia- 
tion precess. Both the technical and cost/price evaluators 
are attempting to gather information in order to complete the 
evaluation process and the contracting personnel are attempting 
to finalize the terms and conditions of the resultant contract. 
The evaluators always want and can use additional information 
and usually are forced to complete the evaluation based on 
incomplete data. At the end of the negotiation process, the 
NFAS activities interviewed request revised proposals to be 
received by a common cutoff date. Then the revised proposals 


are used to complete the evaluation process. Previously 
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unacceptable offerors that were capable of being made accept- 
able are now categorized as either acceptable or not accept- 
able. A final competitive range determination is made. The 
contracting personnel review the proposals of all offerors in 
the competitive range to ensure there are no unacceptable 
terms and conditions. All unsuccessful offerors are notified. 
ae | 

The remaining offerors in the competitive range are 
then given one final opportunity to submit revisions which 
again must be received by a common cutoff date. This is the 
Best and Final (B&F) request and must be clearly identified 
as such. Offerors have a right to know that negotiations have 
concluded, that B&F proposals are being called for and the 
common cutoff date for receipt. [16:3-29] 

In less complex requirements, the submission of revised 
proposals and the request for B&F proposals are compressed into 
one request for B&F proposals. However, there is nothing to 
prevent an offeror from suddenly taking exception to a parti- 
cular term or condition or revising a previously acceptable 
proposal in a way that makes it unacceptable. These problems 
may not be able to be resolved without negotiating with the 
offeror but if negotiations are conducted with one offeror, 
they must be conducted with all offerors. Therefore, it may 
be in the best interest of the Government to ask for another 
round of B&F proposals. This leads to charges of auctioning 
and delays the award, but is permitted where justified or 
necessary. [40:7] 
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H. FINAL EVALUATIONS AND DOCUMENTATION 

The final evaluation shall take into account the results 
of the oral/written negotiations and any proposal revisions 
submitted by offerors as a part of their B&F proposals. The 
offeror may respond to a notification of a weakness, omission, 
or deficiency in one of three ways. The offeror may disagree 
that a deficiency exists, furnish rebuttal information to 
substantiate the original position, and stand on the proposal 
as submitted. The offeror may agree that a deficiency exists 
but, because of previously considered trade-offs, the correc- 
tion of the deficiency may cause loss of some other capability 
considered to be more valuable and determine to propose with 
the noted deficiency. Finally, the offeror may agree that 
a deficiency exists and furnish a correction. [14:47] 

The final evaluation is normally conducted as an update 
of the first evaluation and not a totally new evaluation. 
Care must be taken to evaluate the impact of revision to pro- 
posals not only in the area of the original deficiency but in 
all areas of the proposal. The final documentation is in the 
form of a second part of the TER. The documentation must 
Clearly reflect the effect of negotiations and explain the 
basis for any scoring changes that were made [20:13-6]. The 
final scoresand documentation must reflect the integrated 
collective judgment of the TEP as to the competitive standings 


of the offerors overall [12:4-13]. 
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mee ULTIMATE RESPONSIBILITY FOR SOURCE SELECTION 

In any requirement where evaluation factors other than 
price are used and award is based on the technical competency 
of contractors, the primary effort in source selection is made 
by the TEP members. They alone are presumed capable of con- 
ducting the review of proposals and recommending and documenting 
the award. However, the ultimate responsibility for choice of 
a contractor remains with the contracting officer, the only 
person who can legally bind the Government to a contract. The 
law, regulations and sensible practices all require that, in 
the last analysis, the contracting officer must weight all the 
evidence, recommendations, and other documentation and assume 
the full weight of the ultimate choice. [2:51] 

Problems usually result when two or more proposals are 
evaluated very close to each other. The proposals may be 
either “essentially technically equal" or a "significant 
difference" may exist. The contracting officer may make a 
formal determination backed by factual data that the proposals 
are essentially technically equal. This is in recognition 
that the proposal evaluation is sensitive only to a certain 
degree. Of the proposals determined to be equal, award is then 
Made to the lowest priced proposal [59]. Whether a signifi- 
cant difference exists depends on the facts and circumstances 
of each procurement [61]. 

Depending on the contract type and dollar value of the 
proposed contract, several levels of review may be required 


prior to award. The contracting officer may try to expedite 
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the process, but he has little or no control over the 
reviewers. The time for the reviews is subject to the work- 


load of the reviewers. [5:4-3] 


foeeuNSoUCCESSFUL OFFERORS 
fae NOtiIfLication 

There are three basic reasons why and three basic 
times when an offeror may be notified that his proposal is 
unsuccessful. A proposal may be unacceptable, acceptable 
but outside the competitive range or acceptable but simply 
not selected for award. The offeror must be promptly noti- 
fied that his proposal is no longer being considered for 
contract award. A notification stating that the proposal was 
unacceptable shall cite the appropriate reasons. [12:4-15] 

gee Debr jefing 

The basic purpose of a debriefing is to furnish 
information that should permit the offeror to submit a better 
proposal in the future. Government personnel may also learn 
yvyaluable information, therefore, a debriefing is often in the 
best interests of the Government. Any debriefing is to be 
conducted by the contracting officer with necessary support 
provided by the TEP members. The debriefings are not to be 
conducted until after contract award and with only one offeror 
at a time. The debriefing generally consists of a general 
discussion of the offeror's proposal, its weak and strong 
points in relation to the requirements of the RFP and not 


relative to the other proposals [14:66]. It is imperative 
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that the TEP members be cautioned not to discuss other pro- 
posals and to refrain from implying that the evaluation was 
conducted using any factors other than those set forth in the 
Pep [54]. 

The offerors should be encouraged to submit written 
questions to the Government prior to the actual debriefing. 
This will allow a much more efficient use of time during the 
debriefing. Of utmost importance in debriefing discussions 
is the use of tact. The TEP member should answer appropriate 
questions pertaining to the defects found in the proposal in 
a straightforward manner. At the conclusion of the debriefing, 
the contracting officer should develop a record of the de- 
Briefing including the questions asked, the answers given and 


any other apporpriate data. [14:67] 
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VII. SUMMARY AND RECOMMENDATIONS 


A. SUMMARY 

As an aid to the technical evaluator, a model of the 
major phases and significant events in the proposal evalua- 
tion and source selection process is provided as figure 10. 
Various administrative and review events internal to the 
SentG@racting activity are not included. Individual procure- 
ment actions may have minor variations from the model. For 
example, award on initial offers would eliminate several 
Significant events. 

Although it is obviously better for a technical evaluator 
to become involved in a particular acquisition as early as 
possible, he may enter the process at any one of the five 
phases. Most often, they enter in the early stages of the 
pre=solicitation phase. It is imperative that the technical 
evaluator understand what phase he 1s in and the resulting 
constraints with each phase. 

l. Phase I Pre-procurement Request 

The importance of an informed start cannot be over 
emphasized. Initially, technical personnel should be encour- 
aged to describe the desired services free of constraints in 
order to promote innovation and the identification of needs. 
Open communication with private industry and contracting 
personnel is important in order to gather the necessary 
information. However, as the requirement starts to take shape, 
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Model of Major Events in Source Selection 


Phase [I Phase ITI Phase III 
Pre-procurement request Pre-solicitation Solicitation 


Prepare initial 
Statement of Work (SOW) Receive PR Issue RFP 


Select initial 
evaluation factors 


Select contract 
type 


Pre-proposal 


conference 





Appoint Technical 


dget 
Prepare budge Evaluation Panel (TEP) 


estimate 


Amend RFP 





Issure Procurement 


th 
Request (PR) Prepare smoo 


SOW 


Select final 
evaluation factors 


Prepare Source 
Selection Plan (SSP) 


Prepare Independent 
Cost Estimate (ICE) 


Develop source 
LESt 


Prepare Request 
for Proposal (RFP) Figure 10 





100 





Model of Major Events 
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communication with private industry becomes more constrained 
in order not to give a competitive advantage to any particular 
Soentractor. 

AS soon aS a Significant requirement for services is 
firm enough to provide a basic description, contracting 
personnel should be contacted to discuss the potential for 
competition, type of contract, and source selection method. 

If the source selection method chosen requires the use of 
evaluation factors, contracting personnel should fully explain 
their proper use. The statement of work (SOW) should be 
written with standards that consistently establish a particular 
quality level. Evaluation factors selected must be consistent 
with the SOW and the established standards. The procurement 
request (PR) must also include a budget estimate of the funds 
required. 

2. Phase II Pre-solicitation 

The pre-solicitation phase starts with the receipt of 
the PR by the NFAS activity and ends when the Request for 
Proposal (RFP) is issued. This phase is critical for success- 
ful source selection because of its impact on future events 
and activities. Sufficient time should be provided for 
planning and preparation. The contract type is a major con- 
Sideration of potential offerors. The selection of competent 
and qualified Technical Evaluation Panel (TEP) members influ- 
ences the quality of the evaluations and ultimate source selec- 
tion. The SOW and evaluation factors selected impact the 


ability of potential offerors to submit proposals that meet the 
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needs of the Government. The Source Selection Plan (SSP), 
which includes evaluation factors, weights, scoring system, 
and proposed event milestones, impacts the smoothness and 
objectivity of the source selection. The issuance of the RFP 
sets the terms and conditions of the source selection process 
for each procurement action, which cannot be changed unless 
the RFP is revised. [73-6] 
oe Phase III SOl1 Cheat ron 

The solicitation phase is normally the most stable of 
the five phases. However, the pre-proposal conference and 
other communication with contractors provide an opportunity to 
prevent potential problems that may arise in later phases. 
The solicitation document may require amendment or the SSP may 
need to be revised. 

4. Phase IV Evaluation 

The evaluation phase begins with the receipt of pro- 
posals but does not have a natural ending. Evaluations may 
Beerinue right up until the award decision is finalized. 
Grossly deficient or clearly inferior proposals may be elimi- 
nated at any time from further consideration. The evaluation 
maace nas a dual purpose of ranking the proposals received 
and determining appropriate negotiation objectives. A com- 
plete record is maintained and made a part of the contract 


file. 
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5. Phase V Source Selection 

The source selection phase starts with the completion 
of the initial proposal evaluation. Several important decisions 
are necessary in the sequence leading up to the final source 
selection decision. The competitive range determination is a 
decision to limit further negotiations to those proposals con- 
Sidered to stand a reasonable chance of subsequent contract 
award. Another major decision is the extent to which written 
or oral negotiations should be conducted to give the offerors 
the opportunity to clarify, correct or improve their respective 
proposals. These decisions greatly affect the final source 
selection decision and provide fertile areas for contractor 


dissatisfaction and protest. 


B. RECOMMENDATIONS 
1. Simple Multi-attribute Rating Technique (SMART) 

It is recommended that the SMART method of evaluation 
be reviewed for adoption by the Naval Supply Systems Command. 
Because SMART is specifically designed for the non-professional 
analyst, it is uniquely suited to proposal evaluation at NFAS 
activities. Most NFAS ee eee currently use a similar 
system and could easily adapt to SMART. A single, well docu- 
Mented technique such as SMART would improve the quality of 
evaluations and provide consistency within and between 


activities. 
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2. Source Selection Plan (SSP) 

It is recommended that NFAS activities be encouraged 
to use formal Source Selection Plans (SSP) with established 
event milestones when evaluation factors are being used. This 
would encourage a greater effort in the planning states and 
aid in the administration of the proposal evaluation and 
source selection process. It would also promote coordination 
between the statement of work and the evaluation factors. 

3. Summary of Rules 

It is recommended that a more complete summary of the 
rules pertaining to proposal evaluation, source selection, 
and service contracting be prepared and maintained by the 
Office of Legal Counsel, Naval Supply Systems Command. The 
potential for protest can be reduced by appropriate planning 
if predecents set by GAO are known to field technical and 
contracting personnel. Preparation and review by counsel is 
necessary to provide validity and to keep the summary current. 

4. Guide for Technical Personnel 

It is recommended that the Naval Material Command issue 
a guide for technical personnel in order to assist them in 
their duties as proposal evaluators. A need for guidance and 
training was universally recognized by all individuals inter- 
viewed at NFAS activities. A guide was considered more desir- 
able than a training course because it can be more widely dis- 


tributed. This guide should contain essentially the same 
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information as contained in this research effort, but ina 


compressed, easy to read form. 
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